Methods for providing access to recordings of game play and generation of custom camera views of the game play

ABSTRACT

A computer-implemented method is provided. The method includes executing a game application on one or more servers of a data center. The game application is for a game and the game is played by a first user of a first client device remote to the data center. The one or more servers interfaced with one or more encoders for compressing interactive video from the game application responsive to input from the first client device and streaming of the interactive video in a compressed format to the first client device for decompression and rendering to a display of the first client device. The method includes storing, at the data center, a recording of at least part of the game played by the first user using the first client device. The method includes storing, at the data center, state data for at least part of the game played by the first user using the first client device. The method includes generating a replay of the recording responsive to input from a second user device. The replay is generated with a different camera view from a camera view in the recording. The replay is executed using as input the state data.

CLAIM OF PRIORITY

This application is a Continuation of U.S. patent application Ser. No.17/002,561, filed on Aug. 25, 2020, (U.S. Pat. No. 11,077,370, issued onAug. 3, 2021), entitled “Methods for Providing Access to Recordings ofGame Play and Generation of Custom Camera Views of the Game Play”, whichis a further continuation of U.S. patent application Ser. No.16/412,317, filed on May 14, 2019, (U.S. Pat. No. 10,751,620, issued onAug. 25, 2020), entitled “System for Combining Recorded ApplicationState With Application Streaming Interactive Video Output”, which is aContinuation of U.S. patent application Ser. No. 14/264,492, filed onApr. 29, 2014 (U.S. Pat. No. 10,286,315, issued on May 14, 2019),entitled “System for Combining Recorded Application State WithApplication Streaming Interactive Video Output”, which is a Continuationof U.S. patent application Ser. No. 11/999,472, filed on Dec. 5, 2007,(U.S. Pat. No. 8,832,772, issued on Sep. 9, 2014), entitled “System forCombining Recorded Application State With Application StreamingInteractive Video Output”, of which are all herein incorporated byreference.

TECHNICAL FIELD

The present disclosure relates generally to the field of data processingsystems that improve a users' ability to manipulate and access audio andvideo media.

BACKGROUND

Recorded audio and motion picture media has been an aspect of societysince the days of Thomas Edison. At the start of the 20^(th) centurythere was wide distribution of recorded audio media (cylinders andrecords) and motion picture media (nickelodeons and movies), but bothtechnologies were still in their infancy. In the late 1920s motionpictures were combined with audio on a mass-market basis, followed bycolor motion pictures with audio. Radio broadcasting gradually evolvedinto a largely advertising-supported form of broadcast mass-market audiomedia. When a television (TV) broadcast standard was established in themid-1940s, television joined radio as a form of broadcast mass-marketmedia bringing previously recorded or live motion pictures into thehome.

By the middle of the 20th century, a large percentage of US homes hadphonograph record players for playing recorded audio media, a radio toreceive live broadcast audio, and a television set to play livebroadcast audio/video (A/V) media. Very often these 3 “media players”(record player, radio and TV) were combined into one cabinet sharingcommon speakers that became the “media center” for the home. Althoughthe media choices were limited to the consumer, the media “ecosystem”was quite stable. Most consumers knew how to use the “media players” andwere able to enjoy the full extent of their capabilities. At the sametime, the publishers of the media (largely the motion picture andtelevisions studios, and the music companies) were able to distributetheir media both to theaters and to the home without suffering fromwidespread piracy or “second sales”, i.e., the resale of used media.Typically publishers do not derive revenue from second sales, and assuch, it reduces revenue that publishers might otherwise derive from thebuyer of used media for new sales. Although there certainly were usedrecords sold during the middle of the 20^(th) century, such sales didnot have a large impact on record publishers because, unlike a motionpicture or video program—which is typically watched once or only a fewtimes by an adult—a music track may be listened to hundreds or eventhousands of times. So, music media is far less “perishable” (i.e., ithas lasting value to an adult consumer) than motion picture/video media.Once a record was purchased, if the consumer liked the music, theconsumer was likely to keep it a long time.

From the middle of the 20^(th) century through the present day, themedia ecosystem has undergone a series of radical changes, both to thebenefit and the detriment of consumers and publishers. With thewidespread introduction of audio recorders, especially cassette tapeswith high-quality stereo sound, there certainly was a higher degree ofconsumer convenience. But it also marked the beginning of what is now awidespread practice with consumer media: piracy. Certainly, manyconsumers used the cassette tapes for taping their own records purelyfor convenience, but increasingly consumers (e.g., students in adormitory with ready access to each others' record collections) wouldmake pirated copies. Also, consumers would tape music played over theradio rather than buying a record or tape from the publisher.

The advent of the consumer VCR led to even more consumer convenience,since now a VCR could be set to record a TV show which could be watchedat a later time, and it also led to the creation of the video rentalbusiness, where movies as well as TV programming could be accessed on an“on demand” basis. The rapid development of mass-market home mediadevices since the mid-1980s has led to an unprecedented level of choiceand convenience for the consumer, and also has led to a rapid expansionof the media publishing market.

Today, consumers are faced with a plethora of media choices as well as aplethora of media devices, many of which are tied to particular forms ofmedia or particular publishers. An avid consumer of media may have astack of devices connected to TVs and computers in various rooms of thehouse, resulting in a “rat's nest” of cables to one or more TV setsand/or personal computers (PCs) as well as a group of remote controls.(In the context of the present application, the term “personal computer”or “PC” refers to any sort of computer suitable for us in the home oroffice, including a desktop, a Macintosh® or other non-Windowscomputers, Windows-compatible devices, UNIX variations, laptops, etc.)These devices may include a video game console, VCR, DVD player, audiosurround-sound processor/amplifier, satellite set-top box, cable TVset-top box, etc. And, for an avid consumer, there may be multiplesimilar-function devices because of compatibility issues. For example, aconsumer may own both a HD-DVD and a Blu-ray DVD player, or both aMicrosoft Xbox® and a Sony Playstation® video game system. Indeed,because of incompatibility of some games across versions of gameconsoles, the consumer may own both an XBox and a later version, such asan Xbox 360®. Frequently, consumers are befuddled as to which videoinput and which remote to use. Even after a disc is placed into thecorrect player (e.g., DVD, HD-DVD, Blu-ray, Xbox or Playstation), thevideo and audio input is selected for that the device, and the correctremote control is found, the consumer is still faced with technicalchallenges. For example, in the case of a wide-screen DVD, the user mayneed to first determine and then set the correct aspect ratio on his TVor monitor screen (e.g., 4:3, Full, Zoom, Wide Zoom, Cinema Wide, etc.).Similarly, the user may need to first determine and then set the correctaudio surround sound system format (e.g., AC-3, Dolby Digital, DTS,etc.). Often times, the consumer is unaware that they may not beenjoying the media content to the full capability of their television oraudio system (e.g., watching a movie squashed at the wrong aspect ratio,or listening to audio in stereo rather than in surround sound).

Increasingly, Internet-based media devices have been added to the stackof devices. Audio devices like the Sonos® Digital Music system streamaudio directly from the Internet. Likewise, devices like the Slingbox™entertainment player record video and stream it through a home networkor out through the Internet where it can be watched remotely on a PC.And Internet Protocol Television (IPTV) services offer cable TV-likeservices through Digital Subscriber Line (DSL) or other home Internetconnections. There have also been recent efforts to integrate multiplemedia functions into a single device, such as the Moxi® Media Center andPCs running Windows XP Media Center Edition. While each of these devicesoffers an element of convenience for the functions that it performs,each lacks ubiquitous and simple access to most media. Further, suchdevices frequently cost hundreds of dollars to manufacture, oftenbecause of the need for expensive processing and/or local storage.Additionally, these modern consumer electronic devices typically consumea great deal of power, even while idle, which means they are expensiveover time and wasteful of energy resources. For example, a device maycontinue to operate if the consumer neglects to turn it off or switchesto a different video input. And, because none of the devices is acomplete solution, it must be integrated with the other stack of devicesin the home, which still leaves the user with a rat's nest of wires anda sea of remote controls.

Furthermore, when many newer Internet-based devices do work properly,they typically offer media in a more generic form than it mightotherwise be available. For example, devices that stream video throughthe Internet often stream just the video material, not the interactive“extras” that often accompany DVDs, like the “making of” videos, games,or director's commentary. This is due to the fact that frequently theinteractive material is produced in a particular format intended for aparticular device that handles interactivity locally. For example, eachof DVD, HD-DVDs and Blu-ray discs have their own particular interactiveformat. Any home media device or local computer that might be developedto support all of the popular formats would require a level ofsophistication and flexibility that would likely make it prohibitivelyexpensive and complex for the consumer to operate.

Adding to the problem, if a new format were introduced later in thefuture the local device may not have the hardware capability to supportthe new format, which would mean that the consumer would have topurchase an upgraded local media device. For example, ifhigher-resolution video or stereoscopic video (e.g., one video streamfor each eye) were introduced at a later date, the local device may nothave the computational capability to decode the video, or it may nothave the hardware to output the video in the new format (e.g., assumingstereoscopy is achieved through 120 fps video synchronized withshuttered glasses, with 60 fps delivered to each eye, if the consumer'svideo hardware can only support 60 fps video, this option would beunavailable absent an upgraded hardware purchase).

The issue of media device obsolescence and complexity is a seriousproblem when it comes to sophisticated interactive media, especiallyvideo games.

Modern video game applications are largely divided into four majornon-portable hardware platforms: Sony PlayStation® 1, 2 and 3 (PS1, PS2,and PS3); Microsoft Xbox® and Xbox 360®; and Nintendo Gamecube® andWii™; and PC-based games. Each of these platforms is different than theothers so that games written to run on one platform usually do not runon another platform. There may also be compatibility problems from onegeneration of device to the next. Even though the majority of softwaregame developers create software games that are designed independent of aparticular platform, in order to run a particular game on a specificplatform a proprietary layer of software (frequently called a “gamedevelopment engine”) is needed to adapt the game for use on a specificplatform. Each platform is sold to the consumer as a “console” (i.e., astandalone box attached to a TV or monitor/speakers) or it is a PCitself. Typically, the video games are sold on optical media such as aBlu-ray DVD, DVD-ROM or CD-ROM, which contains the video game embodiedas a sophisticated real-time software application. As home broadbandspeeds have increased, video games are becoming increasingly availablefor download.

The specificity requirements to achieve platform-compatibility withvideo game software is extremely exacting due to the real-time natureand high computational requirements of advanced video games. Forexample, one might expect full game compatibility from one generation tothe next of video games (e.g., from XBox to XBox 360, or fromPlaystation 2 (“PS2”) to Playstation 3 (“PS3”), just as there is generalcompatibility of productivity applications (e.g., Microsoft Word) fromone PC to another with a faster processing unit or core. However, thisis not the case with video games. Because the video game manufacturerstypically are seeking the highest possible performance for a given pricepoint when a video game generation is released, dramatic architecturalchanges to the system are frequently made such that many games writtenfor the prior generation system do not work on the later generationsystem. For example, XBox was based upon the x86-family of processors,whereas XBox 360 was based upon a PowerPC-family.

Techniques can be utilized to emulate a prior architecture, but giventhat video games are real-time applications, it is often unfeasible toachieve the exact same behavior in an emulation. This is a detriment tothe consumer, the video game console manufacturer and the video gamesoftware publisher. For the consumer, it means the necessity of keepingboth an old and new generation of video game consoles hooked up to theTV to be able to play all games. For the console manufacturer it meanscost associated with emulation and slower adoption of new consoles. Andfor the publisher it means that multiple versions of new games may haveto be released in order to reach all potential consumers—not onlyreleasing a version for each brand of video game (e.g., XBox,Playstation), but often a version for each version of a given brand(e.g., PS2 and PS3). For example, a separate version of Electronic Arts'“Madden NFL 08” was developed for XBox, XBox 360, PS2, PS3, Gamecube,Wii, and PC, among other platforms.

Portable devices, such as cellular (“cell”) phones and portable mediaplayers also present challenges to game developers. Increasingly suchdevices are connected to wireless data networks and are able to downloadvideo games. But, there are a wide variety of cell phones and mediadevices in the market, with a wide range of different displayresolutions and computing capabilities. Also, because such devicestypically have power consumption, cost and weight constraints, theytypically lack advanced graphics acceleration hardware like a GraphicsProcessing Unit (“GPU”), such as devices made by NVIDIA of Santa Clara,Calif. Consequently, game software developers typically develop a givengame title simultaneously for many different types of portable devices.A user may find that a given game title is not available for hisparticular cell phone or portable media player.

In the case of home game consoles, hardware platform manufacturerstypically charge a royalty to the software game developers for theability to publish a game on their platform. Cell phone wirelesscarriers also typically charge a royalty to the game publisher todownload a game into the cell phone. In the case of PC games, there isno royalty paid to publish games, but game developers typically facehigh costs due to the higher customer service burden to support the widerange of PC configurations and installation issues that may arise. Also,PCs typically present less barriers to the piracy of game software sincethey are readily reprogrammable by a technically-knowledgeable user andgames can be more easily pirated and more easily distributed (e.g.,through the Internet). Thus, for a software game developer, there arecosts and disadvantages in publishing on game consoles, cell phones andPCs.

For game publishers of console and PC software, costs do not end there.To distribute games through retail channels, publishers charge awholesale price below the selling price for the retailer to have aprofit margin. The publisher also typically has to pay the cost ofmanufacturing and distributing the physical media holding the game. Thepublisher is also frequently charged a “price protection fee” by theretailer to cover possible contingencies such as where the game does notsell, or if the game's price is reduced, or if the retailer must refundpart or all of the wholesale price and/or take the game back from abuyer. Additionally, retailers also typically charge fees to publishersto help market the games in advertising flyers. Furthermore, retailersare increasingly buying back games from users who have finished playingthem, and then sell them as used games, typically sharing none of theused game revenue with the game publisher. Adding to the cost burdenplaced upon game publishers is the fact that games are often pirated anddistributed through the Internet for users to download and make freecopies.

As Internet broadband speeds have been increasing and broadbandconnectivity has become more widespread in the US and worldwide,particularly to the home and to Internet “cafes” whereInternet-connected PCs are rented, games are increasingly beingdistributed via downloads to PCs or consoles. Also, broadbandconnections are increasingly used for playing multiplayer and massivelymultiplayer online games (both of which are referred to in the presentdisclosure by the acronym “MMOG”). These changes mitigate some of thecosts and issues associated with retail distribution. Downloading onlinegames addresses some of the disadvantages to game publishers in thatdistribution costs typically are less and there are little or no costsfrom unsold media. But downloaded games are still subject to piracy, andbecause of their size (often many gigabytes in size) they can take avery long time to download. In addition, multiple games can fill upsmall disk drives, such as those sold with portable computers or withvideo game consoles. However, to the extent games or MMOGs require anonline connection for the game to be playable, the piracy problem ismitigated since the user is usually required to have a valid useraccount. Unlike linear media (e.g., video and music) which can be copiedby a camera shooting video of the display screen or a microphonerecording audio from the speakers, each video game experience is unique,and can not be copied using simple video/audio recording. Thus, even inregions where copyright laws are not strongly enforced and piracy isrampant, MMOGs can be shielded from piracy and therefore a business canbe supported. For example, Vivendi SA's “World of Warcraft” MMOG hasbeen successfully deployed without suffering from piracy throughout theworld. And many online or MMOG games, such as Linden Lab's “Second Life”MMOG generate revenue for the games' operators through economic modelsbuilt into the games where assets can be bought, sold, and even createdusing online tools. Thus, mechanisms in addition to conventional gamesoftware purchases or subscriptions can be used to pay for the use ofonline games.

While piracy can be often mitigated due to the nature of online orMMOGs, online game operator still face remaining challenges. Many gamesrequire substantial local (i.e., in-home) processing resources foronline or MMOGs to work properly. If a user has a low performance localcomputer (e.g., one without a GPU, such as a low-end laptop), he may notbe able to play the game. Additionally, as game consoles age, they fallfurther behind the state-of-the-art and may not be able to handle moreadvanced games. Even assuming the user's local PC is able to handle thecomputational requirements of a game, there are often installationcomplexities. There may be driver incompatibilities (e.g., if a new gameis downloaded, it may install a new version of a graphics driver thatrenders a previously-installed game, reliant upon an old version of thegraphics driver, inoperable). A console may run out of local disk spaceas more games are downloaded. Complex games typically receive downloadedpatches over time from the game developer as bugs are found and fixed,or if modifications are made to the game (e.g., if the game developerfinds that a level of the game is too hard or too easy to play). Thesepatches require new downloads. But sometimes not all users completedownloading of all the patches. Other times, the downloaded patchesintroduce other compatibility or disk space consumption issues.

Also, during game play, large data downloads may be required to providegraphics or behavioral information to the local PC or console. Forexample, if the user enters a room in a MMOG and encounters a scene or acharacter made up of graphics data or with behaviors that are notavailable on the user's local machine, then that scene or character'sdata must be downloaded. This may result in a substantial delay duringgame play if the Internet connection is not fast enough. And, if theencountered scene or character requires storage space or computationalcapability beyond that of the local PC or console, it can create asituation where the user can not proceed in the game, or must continuewith reduced-quality graphics. Thus, online or MMOG games often limittheir storage and/or computational complexity requirements.Additionally, they often limit the amount of data transfers during thegame. Online or MMOG games may also narrow the market of users that canplay the games.

Furthermore, technically-knowledgeable users are increasinglyreverse-engineering local copies of games and modifying the games sothat they can cheat. The cheats maybe as simple as making a button pressrepeat faster than is humanly possible (e.g., so as to shoot a gun veryrapidly). In games that support in-game asset transactions the cheatingcan reach a level of sophistication that results in fraudulenttransactions involving assets of actual economic value. When an onlineor MMOGs economic model is based on such asset transactions, this canresult in substantial detrimental consequences to the game operators.

The cost of developing a new game has grown as PCs and consoles are ableto produce increasingly sophisticated games (e.g., with more realisticgraphics, such as real-time ray-tracing, and more realistic behaviors,such as real-time physics simulation). In the early days of the videogame industry, video game development was a very similar process toapplication software development; that is, most of the development costwas in the development of the software, as opposed to the development ofthe graphical, audio, and behavioral elements or “assets”, such as thosethat may be developed for a motion picture with extensive specialeffects. Today, many sophisticated video game development efforts moreclosely resemble special effects-rich motion picture development thansoftware development. For instance, many video games provide simulationsof 3-D worlds, and generate increasingly photorealistic (i.e., computergraphics that seem as realistic as live action imagery shotphotographically) characters, props, and environments. One of the mostchallenging aspects of photorealistic game development is creating acomputer-generated human face that is indistinguishable from a liveaction human face. Facial capture technologies such Contour™ RealityCapture developed by Mova of San Francisco, Calif. captures and tracksthe precise geometry of a performer's face at high resolution while itis in motion. This technology allows a 3D face to be rendered on a PC orgame console that is virtually indistinguishable from a captured liveaction face. Capturing and rendering a “photoreal” human face preciselyis useful in several respects. First, highly recognizable celebrities orathletes are often used in video games (often hired at a high cost), andimperfections may be apparent to the user, making the viewing experiencedistracting or unpleasant. Frequently, a high degree of detail isrequired to achieve a high degree of photorealism—requiring therendering of a large number of polygons and high-resolution textures,potentially with the polygons and/or textures changing on aframe-by-frame basis as the face moves.

When high polygon-count scenes with detailed textures change rapidly,the PC or game console supporting the game may not have sufficient RAMto store enough polygon and texture data for the required number ofanimation frames generated in the game segment. Further, the singleoptical drive or single disk drive typically available on a PC or gameconsole is usually much slower than the RAM, and typically can not keepup with the maximum data rate that the GPU can accept in renderingpolygons and textures. Current games typically load most of the polygonsand textures into RAM, which means that a given scene is largely limitedin complexity and duration by the capacity of the RAM. In the case offacial animation, for example, this may limit a PC or a game console toeither a low resolution face that is not photoreal, or to a photorealface that can only be animated for a limited number of frames, beforethe game pauses, and loads polygons and textures (and other data) formore frames.

Watching a progress bar move slowly across the screen as a PC or consoledisplays a message similar to “Loading . . . ” is accepted as aninherent drawback by today's users of complex video games. The delaywhile the next scene loads from the disk (“disk” herein, unlessotherwise qualified, refers to non-volatile optical or magnetic media,as well non-disk media such as semiconductor “Flash” memory) can takeseveral seconds or even several minutes. This is a waste of time and canbe quite frustrating to a game player. As previously discussed, much orall of the delay may be due to the load time for polygon, textures orother data from a disk, but it also may be the case that part of theload time is spent while the processor and/or GPU in the PC or consoleprepares data for the scene. For example, a soccer video game may allowthe players to choose among a large number of players, teams, stadiumsand weather conditions. So, depending on what particular combination ischosen, different polygons, textures and other data (collectively“objects”) may be required for the scene (e.g., different teams havedifferent colors and patterns on their uniforms). It may be possible toenumerate many or all of the various permutations and pre-compute manyor all of the objects in advance and store the objects on the disk usedto store the game. But, if the number of permutations is large, theamount of storage required for all of the objects may be too large tofit on the disk (or too impractical to download). Thus, existing PC andconsole systems are typically constrained in both the complexity andplay duration of given scenes and suffer from long load times forcomplex scenes.

Another significant limitation with prior art video game systems andapplication software systems is that they are increasingly using largedatabases, e.g., of 3D objects such as polygons and textures, that needto be loaded into the PC or game console for processing. As discussedabove, such databases can take a long time to load when stored locallyon a disk. Load time, however, is usually far more severe if thedatabase is stored a remote location and is accessed through theInternet. In such a situation it may take minutes, hours, or even daysto download a large database. Further, such databases are often createda great expense (e.g., a 3D model of a detailed tall-masted sailing shipfor use in a game, movie, or historical documentary) and are intendedfor sale to the local end-user. However, the database is at risk ofbeing pirated once it has been downloaded to the local user. In manycases, a user wants to download a database simply for the sake ofevaluating it to see if it suits the user's needs (e.g., if a 3D costumefor a game character has a satisfactory appearance or look when the userperforms a particular move). A long load time can be a deterrent for theuser evaluating the 3D database before deciding to make a purchase.

Similar issues occur in MMOGs, particularly as games that allow users toutilize increasingly customized characters. For a PC or game console todisplay a character it needs to have access to the database of 3Dgeometry (polygons, textures, etc.) as well as behaviors (e.g., if thecharacter has a shield, whether the shield is strong enough to deflect aspear or not) for that character. Typically, when a MMOG is first playedby a user, a large number of databases for characters are alreadyavailable with the initial copy of the game, which is available locallyon the game's optical disk or downloaded to a disk. But, as the gameprogresses, if the user encounters a character or object whose databaseis not available locally (e.g., if another user has created a customizedcharacter), before that character or object can be displayed, itsdatabase must be downloaded. This can result in a substantial delay ofthe game.

Given the sophistication and complexity of video games, anotherchallenge for video game developers and publishers with prior art videogame consoles, is that it frequently takes 2 to 3 years to develop avideo game at a cost of tens of millions of dollars. Given that newvideo game console platforms are introduced at a rate of roughly onceevery five years, game developers need to start development work onthose games years in advance of the release of the new game console inorder to have video games available concurrently when the new platformis released. Several consoles from competing manufactures are sometimesreleased around the same time (e.g., within a year or two of eachother), but what remains to be seen is the popularity of each console,e.g., which console will produce the largest video game software sales.For example, in a recent console cycle, the Microsoft XBox 360, the SonyPlaystation 3, and the Nintendo Wii were scheduled to be introducedaround the same general timeframe. But years before the introductionsthe game developers essentially had to “place their bets” on whichconsole platforms would be more successful than others, and devote theirdevelopment resources accordingly. Motion picture production companiesalso have to apportion their limited production resources based on whatthey estimate to be the likely success of a movie well in advance of therelease of the movie. Given the growing level of investment required forvideo games, game production is increasingly becoming like motionpicture production, and game production companies routinely devote theirproduction resources based on their estimate of the future success of aparticular video game. But, unlike they motion picture companies, thisbet is not simply based on the success of the production itself; rather,it is predicated on the success of the game console the game is intendedto run on. Releasing the game on multiple consoles at once may mitigatethe risk, but this additional effort increases cost, and frequentlydelays the actual release of the game.

Application software and user environments on PCs are becoming morecomputationally intensive, dynamic and interactive, not only to makethem more visually appealing to users, but also to make them more usefuland intuitive. For example, both the new Windows Vista™ operating systemand successive versions of the Macintosh® operating system incorporatevisual animation effects. Advanced graphics tools such as Maya™ fromAutodesk, Inc., provide very sophisticated 3D rendering and animationcapability which push the limits of state-of-the-art CPUs and GPUs.However, the computational requirements of these new tools create anumber of practical issues for users and software developers of suchproducts.

Since the visual display of an operating system (OS) must work on a widerange of classes of computers—including prior-generation computers nolonger sold, but still upgradeable with the new OS—the OS graphicalrequirements are limited to a large degree by a least common denominatorof computers that the OS is targeted for, which typically includescomputers that do not include a GPU. This severely limits the graphicscapability of the OS. Furthermore, battery-powered portably computers(e.g., laptops) limit the visual display capability since highcomputational activity in a CPU or GPU typically results in higher powerconsumption and shorter battery life. Portable computers typicallyinclude software that automatically lowers processor activity to reducepower consumption when the processor is not utilized. In some computermodels the user may lower processor activity manually. For example,Sony's VGN-SZ280P laptop contains a switch labeled “Stamina” on one side(for low performance, more battery life) and “Speed” on the other (forhigh performance, less battery life). An OS running on a portablecomputer must be able to function usably even in the event the computeris running at a fraction of its peak performance capability. Thus, OSgraphics performance often remains far below the state-of-the-artavailable computational capability.

High-end computationally-intense applications like Maya are frequentlysold with the expectation that they will be used on high-performancePCs. This typically establishes a much higher performance, and moreexpensive and less portable, least common denominator requirement. As aconsequence, such applications have a much more limited target audiencethan a general purpose OS (or general purpose productivity application,like Microsoft Office) and typically sell in much lower volume thangeneral purpose OS software or general purpose application software. Thepotential audience is further limited because often times it isdifficult for a prospective user to try out such computationally-intenseapplications in advance. For example, suppose a student wants to learnhow to use Maya or a potential buyer already knowledgeable about suchapplications wants to try out Maya before making the investment in thepurchase (which may well involve also buying a high-end computer capableof running Maya). While either the student or the potential buyer coulddownload, or get a physical media copy of, a demo version of Maya, ifthey lack a computer capable of running Maya to its full potential(e.g., handling a complex 3D scene), then they will be unable to make anfully-informed assessment of the product. This substantially limits theaudience for such high-end applications. It also contributes to a highselling price since the development cost is usually amortized across amuch smaller number of purchases than those of a general-purposeapplication.

High-priced applications also create more incentive for individuals andbusinesses to use pirated copies of the application software. As aresult, high-end application software suffers from rampant piracy,despite significant efforts by publishers of such software to mitigatesuch piracy through various techniques. Still, even when using piratedhigh-end applications, users cannot obviate the need to invest inexpensive state-of-the-art PCs to run the pirated copies. So, while theymay obtain use of a software application for a fraction of its actualretail price, users of pirated software are still required to purchaseor obtain an expensive PC in order to fully utilize the application.

The same is true for users of high-performance pirated video games.Although pirates may get the games at fraction of their actual price,they are still required to purchase expensive computing hardware (e.g.,a GPU-enhanced PC, or a high-end video game console like the XBox 360)needed to properly play the game. Given that video games are typically apastime for consumers, the additional cost for a high-end video gamesystem can be prohibitive. This situation is worse in countries (e.g.,China) where the average annual income of workers currently is quite lowrelative to that of the United States. As a result, a much smallerpercentage of the population owns a high-end video game system or ahigh-end PC. In such countries, “Internet cafes”, in which users pay afee to use a computer connected to the Internet, are quite common.Frequently, such Internet cafes have older model or low-end PCs withouthigh performance features, such as a GPU, which might otherwise enableplayers to play computationally-intensive video games. This is a keyfactor in the success of games that run on low-end PCs, such asVivendi's “World of Warcraft” which is highly successful in China, andis commonly played in Internet cafes there. In contrast, acomputationally-intensive game, like “Second Life” is much less likelyto be playable on a PC installed in a Chinese Internet café. Such gamesare virtually inaccessible to users who only have access tolow-performance PCs in Internet cafes.

Barriers also exist for users who are considering purchasing a videogame and would first like to try out a demonstration version of the gameby downloading the demo through the Internet to their home. A video gamedemo is often a full-fledged version of the game with some featuresdisabled, or with limits placed on the amount of game play. This mayinvolve a long process (perhaps hours) of downloading gigabytes of databefore the game can be installed and executed on either a PC or aconsole. In the case of a PC, it may also involve figuring out whatspecial drivers are needed (e.g., DirectX or OpenGL drivers) for thegame, downloading the correct version, installing them, and thendetermining whether the PC is capable of playing the game. This latterstep may involve determining whether the PC has enough processing (CPUand GPU) capability, sufficient RAM, and a compatible OS (e.g., somegames run on Windows XP, but not Vista). Thus, after a long process ofattempting to run a video game demo, the user may well find out that thevideo game demo can't be possibly played, given the user's PCconfiguration. Worse, once the user has downloaded new drivers in orderto try the demo, these driver versions may be incompatible with othergames or applications the user uses regularly on the PC, thus theinstallation of a demo may render previously operable games orapplications inoperable. Not only are these barriers frustrating for theuser, but they create barriers for video game software publishers andvideo game developers to market their games.

Another problem that results in economic inefficiency has to do with thefact that given PC or game console is usually designed to accommodate acertain level of performance requirement for applications and/or games.For example, some PCs have more or less RAM, slower or faster CPUs, andslower or faster GPUs, if they have a GPUs at all. Some games orapplications make take advantage of the full computing power of a givenPC or console, while many games or applications do not. If a user'schoice of game or application falls short of the peak performancecapabilities of the local PC or console, then the user may have wastedmoney on the PC or console for unutilized features. In the case of aconsole, the console manufacturer may have paid more than was necessaryto subsidize the console cost.

Another problem that exists in the marketing and enjoyment of videogames involves allowing a user to watch others playing games before theuser commits to the purchase of that game. Several prior art approachesexist for the recording of video games for replay at a later time. Forexample, U.S. Pat. No. 5,558,339 teaches recording game stateinformation, including game controller actions, during “gameplay” in thevideo game client computer (owned by the same or different user). Thisstate information can be used at a later time to replay some or all ofthe game action on a video game client computer (e.g., PC or console). Asignificant drawback to this approach is that for a user to view therecorded game, the user must possess a video game client computercapable of playing the game and must have the video game applicationrunning on that computer, such that the gameplay is identical when therecorded game state is replayed. Beyond that, the video game applicationhas to be written in such a way that there is no possible executiondifference between the recorded game and the played back game.

For example, game graphics are generally computed on a frame-by-framebasis. For many games, the game logic sometimes may take shorter orlonger than one frame time to compute the graphics displayed for thenext frame, depending on whether the scene is particularly complex, orif there are other delays that slow down execution (e.g., on a PC,another process may be running that takes away CPU cycles from the gameapplications). In such a game, a “threshold” frame that is computed inslightly less than one frame time (say a few CPU clock cycles less) caneventually occur. When that same scene is computed again using the exactsame game state information, it could easily take a few CPU clock cyclesmore than one frame time (e.g., if an internal CPU bus is slightly outof phase with the an external DRAM bus and it introduces a few CPU cycletimes of delay, even if there is no large delay from another processtaking away milliseconds of CPU time from game processing). Therefore,when the game is played back the frame gets calculated in two frametimes rather than a single frame time. Some behaviors are based on howoften the game calculates a new frame (e.g., when the game samples theinput from the game controllers). While the game is played, thisdiscrepancy in the time reference for different behaviors does notimpact game play, but it can result in the played-back game producing adifferent result. For example, if a basketball's ballistics arecalculated at a steady 60 fps rate, but the game controller input issampled based on rate of computed frames, the rate of computed framesmay be 53 fps when the game was recorded, but 52 fps when the game isreplayed, which can make the difference between whether the basketballis blocked from going into the basket or not, resulting in a differentoutcome. Thus, using game state to record video games requires verycareful game software design to ensure that the replay, using the samegame state information, produces the exact same outcome.

Another prior art approach for recording video game is to simply recordthe video output of a PC or video game system (e.g., to a VCR, DVDrecorder, or to a video capture board on a PC). The video then can berewound and replayed, or alternatively, the recorded video uploaded tothe Internet, typically after being compressed. A disadvantage to thisapproach is that when a 3D game sequence is played back, the user islimited to viewing the sequence from only the point of view from whichthe sequence was recorded. In other words, the user cannot change thepoint of view of the scene.

Further, when compressed video of a recorded game sequence played on ahome PC or game console is made available to other users through theInternet, even if the video is compressed in real-time, it may beimpossible to upload the compressed video in real-time to the Internet.The reason why is because many homes in the world that are connected tothe Internet have highly asymmetric broadband connections (e.g., DSL andcable modem typically have far higher downstream bandwidth than upstreambandwidth). Compressed high resolution video sequences often have higherbandwidths than the upstream bandwidth capacity of the network, makingthem impossible to upload in real-time. Thus, there would be asignificant delay after the game sequence is played (perhaps minutes oreven hours) before another user on the Internet would be able to viewthe game. Although this delay is tolerable in certain situations (e.g.,to watch a game player's accomplishments that occurred at a prior time),it eliminates the ability to watch a game live (e.g., a basketballtournament, played by champion players) or with “instant replay”capability as the game is played live.

Another prior art approach allows a viewer with a television receiver towatch video games live, but only under the control of the televisionproduction crew. Some television channels, in both the US and in othercountries provide video game viewing channels, where the televisionviewing audience is able to watch certain video game users (e.g.,top-rated players playing in tournaments) on video game channels. Thisis accomplished by having the video output of the video game systems(PCs and/or consoles) fed into the video distribution and processingequipment for the television channel. This is not unlike when thetelevision channel is broadcasting a live basketball game in whichseveral cameras provide live feeds from different angles around thebasketball court. The television channel then is able to make use oftheir video/audio processing and effects equipment to manipulate theoutput from the various video game systems. For example, the televisionchannel can overlay text on top of the video from a video game thatindicates the status of different players (just as they might overlaytext during a live basketball game), and the television channel canoverdub audio from a commentator who can discuss the action occurringduring the games. Additionally, the video game output can be combinedwith cameras recording video of the actual players of the games (e.g.,showing their emotional response to the game).

One problem with this approach is that such live video feeds must beavailable to the television channel's video distribution and processingequipment in real-time in order for it to have the excitement of a livebroadcast. As previously discussed, however, this is often impossiblewhen the video game system is running from the home, especially if partof the broadcast includes live video from a camera that is capturingreal-world video of the game player. Further, in a tournament situation,there is a concern that an in-home gamer may modify the game and cheat,as previously described. For these reasons, such video game broadcastson television channels are often arranged with players and video gamesystems aggregated at a common location (e.g., at a television studio orin an arena) where the television production equipment can accept videofeeds from multiple video game systems and potentially live cameras.

Although such prior art video game television channels can provide avery exciting presentation to the television viewing audience that is anexperience akin to a live sporting event, e.g., with the video gameplayers presented as “athletes”, both in terms of their actions in thevideo game world, and in terms of their actions in the real world, thesevideo game systems are often limited to situations where players are inclose physical proximity to one another. And, since television channelsare broadcasted, each broadcasted channel can only show one videostream, which is selected by the television channel's production crew.Because of these limitations and the high cost of broadcast time,production equipment and production crews, such television channelstypically only show top-rated players playing in top tournaments.

Additionally, a given television channel broadcasting a full-screenimage of a video game to the entire television viewing audience showsonly one video game at a time. This severely limits a televisionviewer's choices. For example, a television viewer may not be interestedin the game(s) shown at a given time. Another viewer may only beinterested in watching the game play of a particular player that is notfeatured by the television channel at a given time. In other cases, aviewer may only be interested in watching a how an expert player handlesa particular level in a game. Still other viewers may wish to controlthe viewpoint that a video game is seen from, which is different fromthat chosen by the production team, etc. In short, a television viewermay have a myriad of preferences in watching video games that are notaccommodated by the particular broadcast of a television network, evenif several different television channels are available. For all of theaforementioned reasons, prior art video game television channels havesignificant limitations in presenting video games to television viewers.

Another drawback of prior art video games systems and applicationsoftware systems is that they are complex, and commonly suffer fromerrors, crashes and/or unintended and undesired behaviors (collectively,“bugs”). Although games and applications typically go through adebugging and tuning process (frequently called “Software QualityAssurance” or SQA) before release, almost invariably once the game orapplication is released to a wide audience in the field bugs crop up.Unfortunately, it is difficult for the software developer to identifyand track down many of the bugs after release. It can be difficult forsoftware developers to become aware of bugs. Even when they learn abouta bug, there may only be a limited amount of information available tothem to identify what caused the bug. For example, a user may call up agame developer's customer service line and leave a message stating thatwhen playing the game, the screen started to flash, then changed to asolid blue color and the PC froze. That provides the SQA team with verylittle information useful in tracking down a bug. Some games orapplications that are connected online can sometimes provide moreinformation in certain cases. For example, a “watchdog” process cansometimes be used to monitor the game or application for “crashes”. Thewatchdog process can gather statistics about the status of the game orapplications process (e.g., the status of the stack, of the memoryusage, how far the game or applications has progressed, etc.) when itcrashes and then upload that information to the SQA team via theInternet. But in a complex game or application, such information cantake a very long time to decipher in order to accurately determine whatthe user was doing at the time of the crash. Even then, it may beimpossible to determine what sequence of events led to the crash.

Yet another problem associated with PCs and game consoles is that theyare subject to service issues which greatly inconvenience the consumer.Service issues also impact the manufacturer of the PC or game consolesince they typically are required to send a special box to safely shipthe broken PC or console, and then incur the cost of repair if the PC orconsole is in warranty. The game or application software publisher canalso be impacted by the loss of sales (or online service use) by PCsand/or consoles being in a state of repair.

FIG. 1 illustrates a prior art video gaming system such as a SonyPlaystation® 3, Microsoft Xbox 360®, Nintendo Wii™, Windows-basedpersonal computer or Apple Macintosh. Each of these systems includes acentral processing unit (CPU) for executing program code, typically agraphical processing unit (GPU) for performing advanced graphicaloperations, and multiple forms of input/output (I/O) for communicatingwith external devices and users. For simplicity, these components areshown combined together as a single unit 100. The prior art video gamingsystem of FIG. 1 also is shown including an optical media drive 104(e.g., a DVD-ROM drive); a hard drive 103 for storing video game programcode and data; a network connection 105 for playing multi-player games,for downloading games, patches, demos or other media; a random accessmemory (RAM) 101 for storing program code currently being executed bythe CPU/GPU 100; a game controller 106 for receiving input commands fromthe user during gameplay; and a display device 102 (e.g., a SDTV/HDTV ora computer monitor).

The prior art system shown in FIG. 1 suffers from several limitations.First, optical drives 104 and hard drives 103 tend to have much sloweraccess speeds as compared to that of RAM 101. When working directlythrough RAM 101, the CPU/GPU 100 can, in practice, process far morepolygons per second than is possible when the program code and data isread directly off of hard drive 103 or optical drive 104 due to the factthat RAM 101 generally has much higher bandwidth and does not sufferfrom the relatively long seek delays of disc mechanisms. But only alimited amount of RAM is provided in these prior art systems (e.g.,256-512 Mbytes). Therefore, a “Loading . . . ” sequence in which RAM 101is periodically filled up with the data for the next scene of the videogame is often required.

Some systems attempt to overlap the loading of the program codeconcurrently with the gameplay, but this can only be done when there isa known sequence of events (e.g., if a car is driving down a road, thegeometry for the approaching buildings on the roadside can be loadedwhile the car is driving). For complex and/or rapid scene changes, thistype of overlapping usually does not work. For example, in the casewhere the user is in the midst of a battle and RAM 101 is completelyfilled with data representing the objects within view at that moment, ifthe user moves the view rapidly to the left to view objects that are notpresently loaded in RAM 101, a discontinuity in the action will resultsince there not be enough time to load the new objects from Hard Drive103 or Optical Media 104 into RAM 101.

Another problem with the system of FIG. 1 arises due to limitations inthe storage capacity of hard drives 103 and optical media 104. Althoughdisk storage devices can be manufactured with a relatively large storagecapacity (e.g., 50 gigabytes or more), they still do not provide enoughstorage capacity for certain scenarios encountered in current videogames. For example, as previously mentioned, a soccer video game mightallow the user to choose among dozens of teams, players and stadiumsthroughout the world. For each team, each player and each stadium alarge number of texture maps and environment maps are needed tocharacterize the 3D surfaces in the world (e.g., each team has a uniquejersey, with each requiring a unique texture map).

One technique used to address this latter problem is for the game topre-compute texture and environment maps once they are selected by theuser. This may involve a number of computationally-intensive processes,including decompressing images, 3D mapping, shading, organizing datastructures, etc. As a result, there may be a delay for the user whilethe video game is performing these calculations. On way to reduce thisdelay, in principle, is to perform all of these computations—includingevery permutation of team, player roster, and stadium—when the game wasoriginally developed. The released version of the game would theninclude all of this pre-processed data stored on optical media 104, oron one or more servers on the Internet with just the selectedpre-processed data for a given team, player roster, stadium selectiondownloaded through the Internet to hard drive 103 when the user makes aselection. As a practical matter, however, such pre-loaded data of everypermutation possible in game play could easily be terabytes of data,which is far in excess of the capacity of today's optical media devices.Furthermore, the data for a given team, player roster, stadium selectioncould easily be hundreds of megabytes of data or more. With a homenetwork connection of, say, 10 Mbps, it would take longer to downloadthis data through network connection 105 than it would to compute thedata locally.

Thus, the prior art game architecture shown in FIG. 1 subjects the userto significant delays between major scene transitions of complex games.

Another problem with prior art approaches such as that shown in FIG. 1is that over the years video games tend to become more advanced andrequire more CPU/GPU processing power. Thus, even assuming an unlimitedamount of RAM, video games hardware requirements go beyond the peaklevel of processing power available in these systems. As a result, usersare required to upgrade gaming hardware every few years to keep pace (orplay newer games at lower quality levels). One consequence of the trendto ever more advanced video games is that video game playing machinesfor home use are typically economically inefficient because their costis usually determined by the requirements of the highest performancegame they can support. For example, an XBox 360 might be used to play agame like “Gears of War”, which demands a high performance CPU, GPU, andhundreds of megabytes of RAM, or the XBox 360 might be used to play PacMan, a game from the 1970s that requires only kilobytes of RAM and avery low performance CPU. Indeed, an XBox 360 has enough computing powerto host many simultaneous Pac Man games at once.

Video games machines are typically turned off for most of the hours of aweek. According to a July 2006 Nielsen Entertainment study of activegamers 13 years and older, on average, active gamers spend fourteenhours/week playing console video games, or just 12% of the total hoursin a week. This means that the average video game console is idle 88% ofthe time, which is an inefficient use of an expensive resource. This isparticularly significant given that video game consoles are oftensubsidized by the manufacturer to bring down the purchase price (withthe expectation that the subsidy will be earned back by royalties fromfuture video game software purchases).

Video game consoles also incur costs associated with almost any consumerelectronic device. For instance, the electronics and mechanisms of thesystems need to be housed in an enclosure. The manufacturer needs tooffer a service warranty. The retailer who sells the system needs tocollect a margin on either the sale of the system and/or on the sale ofvideo game software. All of these factors add to the cost of the videogame console, which must either be subsidized by the manufacturer,passed along to the consumer, or both.

In addition, piracy is a major problem for the video game industry. Thesecurity mechanisms utilized on virtually every major video gamingsystem have been “cracked” over the years, resulting in unauthorizedcopying of video games. For example, the Xbox 360 security system wascracked in July 2006 and users are now able to download illegal copiesonline. Games that are downloadable (e.g., games for the PC or the Mac)are particularly vulnerable to piracy. In certain regions of the worldwhere piracy is weakly policed there is essentially no viable market forstandalone video game software because users can buy pirated copies asreadily as legal copies for a tiny fraction of the cost. Also, in manyparts of the world the cost of a game console is such a high percentageof income that even if piracy were controlled, few people could afford astate-of-the-art gaming system.

In addition, the used game market reduces revenue for the video gameindustry. When a user has become tired of a game, they can sell the gameto a store which will resell the game to other users. This unauthorizedbut common practice significantly reduces revenues of game publishers.Similarly, a reduction in sales on the order of 50% commonly occurs whenthere is a platform transition every few years. This is because usersstop buying games for the older platforms when they know that the newerversion platform is about to be released (e.g., when Playstation 3 isabout to be released, users stop buying Playstation 2 games). Combined,the loss of sales and increased development costs associated with thenew platforms can have a very significant adverse impact on theprofitability of game developers.

New game consoles are also very expensive. The Xbox 360, the NintendoWii, and the Sony Playstation 3 all retail for hundreds of dollars. Highpowered personal computer gaming systems can cost up to $8000. Thisrepresents a significant investment for users, particularly consideringthat the hardware becomes obsolete after a few years and the fact thatmany systems are purchased for children.

One approach to the foregoing problems is online gaming in which thegaming program code and data are hosted on a server and delivered toclient machines on-demand as compressed video and audio streamed over adigital broadband network. Some companies such as G-Cluster in Finland(now a subsidiary of Japan's SOFTBANK Broadmedia) currently providethese services online. Similar gaming services have become available inlocal networks, such as those within hotels and offered by DSL and cabletelevision providers. A major drawback of these systems is the problemof latency, i.e., the time it takes for a signal to travel to and fromthe game server, which is typically located in an operator's “head-end”.Fast action video games (also known as “twitch” video games) requirevery low latency between the time the user performs an action with thegame controller and the time the display screen is updated showing theresult of the user action. Low latency is needed so that the user hasthe perception that the game is responding “instantly”. Users may besatisfied with different latency intervals depending on the type of gameand the skill level of the user. For example, 100 ms of latency may betolerable for a slow casual game (like backgammon) or a slow-action roleplaying game, but in a fast action game a latency in excess of 70 or 80ms may cause the user to perform more poorly in the game, and thus isunacceptable. For instance, in a game that requires fast reaction timethere is a sharp decline in accuracy as latency increases from 50 to 100ms.

When a game or application server is installed in a nearby, controllednetwork environment, or one where the network path to the user ispredictable and/or can tolerate bandwidth peaks, it is far easier tocontrol latency, both in terms of maximum latency and in terms of theconsistency of the latency (e.g., so the user observes steady motionfrom digital video streaming through the network). Such level of controlcan be achieved between a cable TV network head-end to a cable TVsubscriber's home, or from a DSL central office to DSL subscriber'shome, or in a commercial office Local Area Network (LAN) environmentfrom a server or a user. Also, it is possible to obtain specially-gradedpoint-to-point private connections between businesses which haveguaranteed bandwidth and latency. But in a game or application systemthat hosts games in a server center connected to the general Internetand then streams compressed video to the user through a broadbandconnection, latency is incurred from many factors, resulting in severelimitations in the deployment of prior art systems.

In a typical broadband-connected home, a user may have a DSL or cablemodem for broadband service. Such broadband services commonly incur asmuch as a 25 ms round-trip latency (and at times more) between theuser's home and the general Internet. In addition, there are round-triplatencies incurred from routing data through the Internet to a servercenter. The latency through the Internet varies based on the route thatthe data is given and the delays it incurs as it is routed. In additionto routing delays, round-trip latency is also incurred due to the speedof light traveling through the optical fiber that interconnects most ofthe Internet. For example, for each 1000 miles, approximately 22 ms isincurred in round-trip latency due to the speed of light through theoptical fiber and other overhead.

Additional latency can occur due to the data rate of the data streamedthrough the Internet. For example, if a user has DSL service that issold as “6 Mbps DSL service”, in practice, the user will probably getless than 5 Mbps of downstream throughput at best, and will likely seethe connection degrade periodically due to various factors such ascongestion during peak load times at the Digital Subscriber Line AccessMultiplexer (DSLAM). A similar issue can occur reducing a the data rateof a cable modem is used for a connection sold as “6 Mbps cable modemservice” to far less than that, if there is congestion in the localshared coaxial cable looped through the neighborhood, or elsewhere inthe cable modem system network. If data packets at a steady rate of 4Mbps are streamed as one-way in User Datagram Protocol (UDP) format froma server center through such connections, if everything is working well,the data packets will pass through without incurring additional latency,but if there is congestion (or other impediments) and only 3.5 Mbps isavailable to stream data to the user, then in a typical situation eitherpackets will be dropped, resulting in lost data, or packets will queueup at the point of congestion, until they can be sent, therebyintroducing additional latency. Different points of congestion havedifferent queuing capacity to hold delayed packets, so in some casespackets that can't make it through the congestion are droppedimmediately. In other cases, several megabits of data are queued up andeventually be sent. But, in almost all cases, queues at points ofcongestion have capacity limits, and once those limits are exceeded, thequeues will overflow and packets will be dropped. Thus, to avoidincurring additional latency (or worse, loss of packets), it isnecessary to avoid exceeding the data rate capacity from the game orapplication server to the user.

Latency is also incurred by the time required to compress video in theserver and decompress video in the client device. Latency is furtherincurred while a video game running on a server is calculating the nextframe to be displayed. Currently available video compression algorithmssuffer from either high data rates or high latency. For example, motionJPEG is an intraframe-only lossy compression algorithm that ischaracterized by low-latency. Each frame of video is compressedindependently of each other frame of video. When a client devicereceives a frame of compressed motion JPEG video, it can immediatelydecompress the frame and display it, resulting in very low latency. Butbecause each frame is compressed separately, the algorithm is unable toexploit similarities between successive frames, and as a resultintraframe-only video compression algorithms suffer from very high datarates. For example, 60 fps (frames per second) 640×480 motion JPEG videomay require 40 Mbps (megabits per second) or more of data. Such highdata rates for such low resolution video windows would be prohibitivelyexpensive in many broadband applications (and certainly for mostconsumer Internet-based applications). Further, because each frame iscompressed independently, artifacts in the frames that may result fromthe lossy compression are likely to appear in different places insuccessive frames. This can results in what appears to the viewer as amoving visual artifacts when the video is decompressed.

Other compression algorithms, such as MPEG2, H.264 or VC9 from MicrosoftCorporation as they are used in prior art configurations, can achievehigh compression ratios, but at the cost of high latency. Suchalgorithms utilize interframe as well as intraframe compression.Periodically, such algorithms perform an intraframe-only compression ofa frame. Such a frame is known as a key frame (typically referred to asan “I” frame). Then, these algorithms typically compare the I frame withboth prior frames and successive frames. Rather than compressing theprior frames and successive frames independently, the algorithmdetermines what has changed in the image from the I frame to the priorand successive frames, and then stores those changes as what are called“B” frames for the changes preceding the I frame and “P” frames for thechanges following the I frame. This results in much lower data ratesthan intraframe-only compression. But, it typically comes at the cost ofhigher latency. An I frame is typically much larger than a B or P frame(often 10 times larger), and as a result, it takes proportionatelylonger to transmit at a given data rate.

Consider, for example, a situation where the I frames are 10× the sizeof B and P frames, and there are 29 B frames+30 P frames=59 interframesfor every single I intraframe, or 60 frames total for each “Group ofFrames” (GOP). So, at 60 fps, there is 1 60-frame GOP each second.Suppose the transmission channel has a maximum data rate of 2 Mbps. Toachieve the highest quality video in the channel, the compressionalgorithm would produce a 2 Mbps data stream, and given the aboveratios, this would result in 2 Megabits (Mb)/(59+10)=30,394 bits perintraframe and 303,935 bits per I frame. When the compressed videostream is received by the decompression algorithm, in order for thevideo to play steadily, each frame needs to decompressed and displayedat a regular interval (e.g., 60 fps). To achieve this result, if anyframe is subject to transmission latency, all of the frames need to bedelayed by at least that latency, so the worst-case frame latency willdefine the latency for every video frame. The I frames introduce thelongest transmission latencies since they are largest, and an entire Iframe would have to be received before the I frame could be decompressedand displayed (or any interframe dependent on the I frame). Given thatthe channel data rate is 2 Mbps, it will take 303,935/2 Mb=145 ms totransmit an I frame.

An interframe video compression system as described above using a largepercentage of the bandwidth of the transmission channel will be subjectto long latencies due to the large size of an I frame relative to theaverage size of a frame. Or, to put it another way, while prior artinterframe compression algorithms achieve a lower average per-frame datarate than intraframe-only compression algorithms (e.g., 2 Mbps vs. 40Mbps), they still suffer from a high peak per-frame data rate (e.g.,303,935*60=18.2 Mbps) because of the large I frames. Bear in mind,though that the above analysis assumes that the P and B frames are allmuch smaller than the I frames. While this is generally true, it is nottrue for frames with high image complexity uncorrelated with the priorframe, high motion, or scene changes. In such situations, the P or Bframes can become as large as I frames (if a P or B frame gets largerthan an I frame, a sophisticated compression algorithm will typically“force” an I frame and replace the P or B frame with an I frame). So, Iframe-sized data rate peaks can occur at any moment in a digital videostream. Thus, with compressed video, when the average video data rateapproaches data rate capacity of the transmission channels (as isfrequently the case, given the high data rate demands for video) thehigh peak data rates from I frames or large P or B frames result in ahigh frame latency.

Of course, the above discussion only characterizes the compressionalgorithm latency created by large B, P or I frames in a GOP. If Bframes are used, the latency will be even higher. The reason why isbecause before a B frame can be displayed, all of the B frames after theB frame and the I frame must be received. Thus, in a group of picture(GOP) sequence such as BBBBBIPPPPPBBBBBIPPPPP, where there are 5 Bframes before each I frame, the first B frame can not be displayed bythe video decompressor until the subsequent B frames and I frame arereceived. So, if video is being streamed at 60 fps (i.e., 16.67ms/frame), before the first B frame can be decompressed, five B framesand the I frame will take 16.67*6=100 ms to receive, no matter how fastthe channel bandwidth is, and this is with just 5 B frames. Compressedvideo sequences with 30 B frames are quite common. And, at a low channelbandwidth like 2 Mbps, the latency impact caused by the size of the Iframe is largely additive to the latency impact due to waiting for Bframes to arrive. Thus, on a 2 Mbps channel, with a large number of Bframes it is quite easy to exceed 500 ms of latency or more using priorart video compression technology. If B frames are not used (at the costof a lower compression ratio for given quality level), the B framelatency is not incurred, but the latency caused by the peak frame sizes,described above, is still incurred.

The problem is exacerbated by very the nature of many video games. Videocompression algorithms utilizing the GOP structure described above havebeen largely optimized for use with live video or motion picturematerial intended for passive viewing. Typically, the camera (whether areal camera, or a virtual camera in the case of a computer-generatedanimation) and scene is relatively steady, simply because if the cameraor scene moves around too jerkily, the video or movie material is (a)typically unpleasant to watch and (b) if it is being watched, usuallythe viewer is not closely following the action when the camera jerksaround suddenly (e.g., if the camera is bumped when shooting a childblowing out the candles on a birthday cake and suddenly jerks away fromthe cake and back again, the viewers are typically focused on the childand the cake, and disregard the brief interruption when the camerasuddenly moves). In the case of a video interview, or a videoteleconference, the camera may be held in a fixed position and not moveat all, resulting in very few data peaks at all. But 3D high actionvideo games are characterized by constant motion (e.g., consider a 3Dracing, where the entire frame is in rapid motion for the duration ofthe race, or consider first-person shooters, where the virtual camera isconstantly moving around jerkily). Such video games can result in framesequences with large and frequent peaks where the user may need toclearly see what is happening during those sudden motions. As such,compression artifacts are far less tolerable in 3D high action videogames. Thus, the video output of many video games, by their nature,produces a compressed video stream with very high and frequent peaks.

Given that users of fast-action video games have little tolerance forhigh latency, and given all of the above causes of latency, to datethere have been limitations to server-hosted video games that streamvideo on the Internet. Further, users of applications that require ahigh degree of interactivity suffer from similar limitations if theapplications are hosted on the general Internet and stream video. Suchservices require a network configuration in which the hosting serversare set up directly in a head end (in the case of cable broadband) orthe central office (in the case of Digital Subscriber Lines (DSL)), orwithin a LAN (or on a specially-graded private connection) in acommercial setting, so that the route and distance from the clientdevice to the server is controlled to minimize latency and peaks can beaccommodated without incurring latency. LANs (typically rated at 100Mbps-1 Gbps) and leased lines with adequate bandwidth typically cansupport peak bandwidth requirements (e.g., 18 Mbps peak bandwidth is asmall fraction of a 100 Mbps LAN capacity).

Peak bandwidth requirements can also be accommodated by residentialbroadband infrastructure if special accommodations are made. Forexample, on a cable TV system, digital video traffic can be givendedicated bandwidth which can handle peaks, such as large I frames. And,on a DSL system, a higher speed DSL modem can be provisioned, allowingfor high peaks, or a specially-graded connection can provisioned whichcan handle a higher data rates. But, conventional cable modem and DSLinfrastructure attached to the general Internet have far less tolerancefor peak bandwidth requirements for compressed video. So, onlineservices that host video games or applications in server centers a longdistance from the client devices, and then stream the compressed videooutput over the Internet through conventional residential broadbandconnections suffer from significant latency and peak bandwidthlimitations—particularly with respect to games and applications whichrequire very low latency (e.g., first person shooters and othermulti-user, interactive action games, or applications requiring a fastresponse time).

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be understood more fully from the detaileddescription that follows and from the accompanying drawings, whichhowever, should not be taken to limit the disclosed subject matter tothe specific embodiments shown, but are for explanation andunderstanding only.

FIG. 1 illustrates an architecture of a prior art video gaming system.

FIGS. 2 a-b illustrate a high level system architecture according to oneembodiment.

FIG. 3 illustrates actual, rated, and required data rates forcommunication between a client and a server.

FIG. 4 a illustrates a hosting service and a client employed accordingto one embodiment.

FIG. 4 b illustrates exemplary latencies associated with communicationbetween a client and hosting service.

FIG. 4 c illustrates a client device according to one embodiment.

FIG. 4 d illustrates a client device according to another embodiment.

FIG. 4 e illustrates an example block diagram of the client device inFIG. 4 c.

FIG. 4 f illustrates an example block diagram of the client device inFIG. 4 d.

FIG. 5 illustrates an example form of video compression which may beemployed according to one embodiment.

FIG. 6 a illustrates an example form of video compression which may beemployed in another embodiment.

FIG. 6 b illustrates peaks in data rate associated with transmitting alow complexity, low action video sequence.

FIG. 6 c illustrates peaks in data rate associated with transmitting ahigh complexity, high action video sequence.

FIGS. 7 a-b illustrate example video compression techniques employed inone embodiment.

FIG. 8 illustrates additional example video compression techniquesemployed in one embodiment.

FIGS. 9 a-c illustrate example techniques employed in one embodiment foralleviating data rate peaks.

FIGS. 10 a-b illustrate one embodiment which efficiently packs imagetiles within packets.

FIGS. 11 a-d illustrate embodiments which employ forward errorcorrection techniques.

FIG. 12 illustrates one embodiment which uses multi-core processingunits for compression.

FIGS. 13 a-b illustrate geographical positioning and communicationbetween hosting services according to various embodiments.

FIG. 14 illustrates exemplary latencies associated with communicationbetween a client and a hosting service.

FIG. 15 illustrates an example hosting service server centerarchitecture.

FIG. 16 illustrates an example screen shot of one embodiment of a userinterface which includes a plurality of live video windows.

FIG. 17 illustrates the user interface of FIG. 16 following theselection of a particular video window.

FIG. 18 illustrates the user interface of FIG. 17 following zooming ofthe particular video window to full screen size.

FIG. 19 illustrates an example collaborative user video data overlaid onthe screen of a multiplayer game.

FIG. 20 illustrates an example user page for a game player on a hostingservice.

FIG. 21 illustrates an example 3D interactive advertisement.

FIG. 22 illustrates an example sequence of steps for producing aphotoreal image having a textured surface from surface capture of a liveperformance.

FIG. 23 illustrates an example user interface page that allows forselection of linear media content.

FIG. 24 is a graph that illustrates the amount of time that elapsesbefore the web page is live versus connection speed.

DESCRIPTION OF EXAMPLE EMBODIMENTS

In the following description specific details are set forth, such asdevice types, system configurations, communication methods, etc., inorder to provide a thorough understanding of the present disclosure.However, persons having ordinary skill in the relevant arts willappreciate that these specific details may not be needed to practice theembodiments described.

FIGS. 2 a-b provide a high-level architecture of two embodiments inwhich video games and software applications are hosted by a hostingservice 210 and accessed by client devices 205 at user premises 211(note that the “user premises” means the place wherever the user islocated, including outdoors if using a mobile device) over the Internet206 (or other public or private network) under a subscription service.The client devices 205 may be general-purpose computers such asMicrosoft Windows- or Linux-based PCs or Apple, Inc. Macintosh computerswith a wired or wireless connection to the Internet either with internalor external display device 222, or they may be dedicated client devicessuch as a set-top box (with a wired or wireless connection to theInternet) that outputs video and audio to a monitor or TV set 222, orthey may be mobile devices, presumably with a wireless connection to theInternet.

Any of these devices may have their own user input devices (e.g.,keyboards, buttons, touch screens, track pads or inertial-sensing wands,video capture cameras and/or motion-tracking cameras, etc.), or they mayuse external input devices 221 (e.g., keyboards, mice, game controllers,inertial sensing wand, video capture cameras and/or motion trackingcameras, etc.), connected with wires or wirelessly. As described ingreater detail below, the hosting service 210 includes servers ofvarious levels of performance, including those with high-powered CPU/GPUprocessing capabilities. During playing of a game or use of anapplication on the hosting service 210, a home or office client device205 receives keyboard and/or controller input from the user, and then ittransmits the controller input through the Internet 206 to the hostingservice 210 that executes the gaming program code in response andgenerates successive frames of video output (a sequence of video images)for the game or application software (e.g., if the user presses a buttonwhich would direct a character on the screen to move to the right, thegame program would then create a sequence of video images showing thecharacter moving to the right). This sequence of video images is thencompressed using a low-latency video compressor, and the hosting service210 then transmits the low-latency video stream through the Internet206. The home or office client device then decodes the compressed videostream and renders the decompressed video images on a monitor or TV.Consequently, the computing and graphical hardware requirements of theclient device 205 are significantly reduced. The client 205 only needsto have the processing power to forward the keyboard/controller input tothe Internet 206 and decode and decompress a compressed video streamreceived from the Internet 206, which virtually any personal computer iscapable of doing today in software on its CPU (e.g., a Intel CorporationCore Duo CPU running at approximately 2 GHz is capable of decompressing720p HDTV encoded using compressors such as H.264 and Windows MediaVC9). And, in the case of any client devices, dedicated chips can alsoperform video decompression for such standards in real-time at far lowercost and with far less power consumption than a general-purpose CPU suchas would be required for a modern PC. Notably, to perform the functionof forwarding controller input and decompressing video, home clientdevices 205 do not require any specialized graphics processing units(GPUs), optical drive or hard drives, such as the prior art video gamesystem shown in FIG. 1 .

As games and applications software become more complex and morephoto-realistic, they will require higher-performance CPUs, GPUs, moreRAM, and larger and faster disk drives, and the computing power at thehosting service 210 may be continually upgraded, but the end user willnot be required to update the home or office client platform 205 sinceits processing requirements will remain constant for a displayresolution and frame rate with a given video decompression algorithm.Thus, the hardware limitations and compatibility issues seen today donot exist in the system illustrated in FIGS. 2 a -b.

Further, because the game and application software executes only inservers in the hosting service 210, there never is a copy of the game orapplication software (either in the form of optical media, or asdownloaded software) in the user's home or office (“office” as usedherein unless otherwise qualified shall include any non-residentialsetting, including, schoolrooms, for example). This significantlymitigates the likelihood of a game or application software beingillegally copied (pirated), as well as mitigating the likelihood of avaluable database that might be use by a game or applications softwarebeing pirated. Indeed, if specialized servers are required (e.g.,requiring very expensive, large or noisy equipment) to play the game orapplication software that are not practical for home or office use, theneven if a pirated copy of the game or application software wereobtained, it would not be operable in the home or office.

In one embodiment, the hosting service 210 provides software developmenttools to the game or application software developers (which refersgenerally to software development companies, game or movie studios, orgame or applications software publishers) 220 which design video gamesso that they may design games capable of being executed on the hostingservice 210. Such tools allow developers to exploit features of thehosting service that would not normally be available in a standalone PCor game console (e.g., fast access to very large databases of complexgeometry (“geometry” unless otherwise qualified shall be used herein torefer to polygons, textures, rigging, lighting, behaviors and othercomponents and parameters that define 3D datasets)).

Different business models are possible under this architecture. Underone model, the hosting service 210 collects a subscription fee from theend user and pays a royalty to the developers 220, as shown in FIG. 2 a. In an alternate implementation, shown in FIG. 2 b , the developers 220collects a subscription fee directly from the user and pays the hostingservice 210 for hosting the game or application content. Theseunderlying principles are not limited to any particular business modelfor providing online gaming or application hosting.

Compressed Video Characteristics

As discussed previously, one significant problem with providing videogame services or applications software services online is that oflatency. A latency of 70-80 ms (from the point a input device isactuated by the user to the point where a response is displayed on thedisplay device) is at the upper limit for games and applicationsrequiring a fast response time. However, this is very difficult toachieve in the context of the architecture shown in FIGS. 2 a and 2 bdue to a number of practical and physical constraints.

As indicated in FIG. 3 , when a user subscribes to an Internet service,the connection is typically rated by a nominal maximum data rate 301 tothe user's home or office. Depending on the provider's policies androuting equipment capabilities, that maximum data rate may be more orless strictly enforced, but typically the actual available data rate islower for one of many different reasons. For example, there may be toomuch network traffic at the DSL central office or on the local cablemodem loop, or there may be noise on the cabling causing droppedpackets, or the provider may establish a maximum number of bits permonth per user. Currently, the maximum downstream data rate for cableand DSL services typically ranges from several hundred Kilobits/second(Kbps) to 30 Mbps. Cellular services are typically limited to hundredsof Kbps of downstream data. However, the speed of the broadband servicesand the number of users who subscribe to broadband services willincrease dramatically over time. Currently, some analysts estimate that33% of US broadband subscribers have a downstream data rate of 2 Mbps ormore. For example, some analysts predict that by 2010, over 85% of USbroadband subscribers will have a data rate of 2 Mbps or more.

As indicated in FIG. 3 , the actual available max data rate 302 mayfluctuate over time. Thus, in a low-latency, online gaming orapplication software context it is sometimes difficult to predict theactual available data rate for a particular video stream. If the datarate 303 required to sustain a given level of quality at given number offrames-per-second (fps) at a given resolution (e.g., 640×480 @ 60 fps)for a certain amount of scene complexity and motion rises above theactual available max data rate 302 (as indicated by the peak in FIG. 3), then several problems may occur. For example, some internet serviceswill simply drop packets, resulting in lost data and distorted/lostimages on the user's video screen. Other services will temporarilybuffer (i.e., queue up) the additional packets and provide the packetsto the client at the available data rate, resulting in an increase inlatency—an unacceptable result for many video games and applications.Finally, some Internet service providers will view the increase in datarate as a malicious attack, such as a denial of service attack (a wellknown technique user by hackers to disable network connections), andwill cut off the user's Internet connection for a specified time period.Thus, the embodiments described herein take steps to ensure that therequired data rate for a video game does not exceed the maximumavailable data rate.

Hosting Service Architecture

FIG. 4 a illustrates an architecture of the hosting service 210according to one embodiment. The hosting service 210 can either belocated in a single server center, or can be distributed across aplurality of server centers (to provide for lower latency connections tousers that have lower latency paths to certain server centers thanothers, to provide for load balancing amongst users, and to provide forredundancy in the case one or more server centers fail). The hostingservice 210 may eventually include hundreds of thousands or evenmillions of servers 402, serving a very large user base. A hostingservice control system 401 provides overall control for the hostingservice 210, and directs routers, servers, video compression systems,billing and accounting systems, etc. In one embodiment, the hostingservice control system 401 is implemented on a distributed processingLinux-based system tied to RAID arrays used to store the databases foruser information, server information, and system statistics. In theforegoing descriptions, the various actions implemented by the hostingservice 210, unless attributed to other specific systems, are initiatedand controlled by the hosting service control system 401.

The hosting service 210 includes a number of servers 402 such as thosecurrently available from Intel, IBM and Hewlett Packard, and others.Alternatively, the servers 402 can be assembled in a customconfiguration of components, or can eventually be integrated so anentire server is implemented as a single chip. Although this diagramshows a small number of servers 402 for the sake of illustration, in anactual deployment there may be as few as one server 402 or as many asmillions of servers 402 or more. The servers 402 may all be configuredin the same way (as an example of some of the configuration parameters,with the same CPU type and performance; with or without a GPU, and ifwith a GPU, with the same GPU type and performance; with the same numberof CPUs and GPUs; with the same amount of and type/speed of RAM; andwith the same RAM configuration), or various subsets of the servers 402may have the same configuration (e.g., 25% of the servers can beconfigured a certain way, 50% a different way, and 25% yet another way),or every server 402 may be different.

In one embodiment, the servers 402 are diskless, i.e., rather thanhaving its own local mass storage (be it optical or magnetic storage, orsemiconductor-based storage such as Flash memory or other mass storagemeans serving a similar function), each server accesses shared massstorage through fast backplane or network connection. In one embodiment,this fast connection is a Storage Area Network (SAN) 403 connected to aseries of Redundant Arrays of Independent Disks (RAID) 405 withconnections between devices implemented using Gigabit Ethernet. As isknown by those of skill in the art, a SAN 403 may be used to combinemany RAID arrays 405 together, resulting in extremely highbandwidth—approaching or potentially exceeding the bandwidth availablefrom the RAM used in current gaming consoles and PCs. And, while RAIDarrays based on rotating media, such as magnetic media, frequently havesignificant seek-time access latency, RAID arrays based on semiconductorstorage can be implemented with much lower access latency. In anotherconfiguration, some or all of the servers 402 provide some or all oftheir own mass storage locally. For example, a server 402 may storefrequently-accessed information such as its operating system and a copyof a video game or application on low-latency local Flash-based storage,but it may utilize the SAN to access RAID Arrays 405 based on rotatingmedia with higher seek latency to access large databases of geometry orgame state information on a less frequent bases.

In addition, in one embodiment, the hosting service 210 employslow-latency video compression logic 404 described in detail below. Thevideo compression logic 404 may be implemented in software, hardware, orany combination thereof (certain embodiments of which are describedbelow). Video compression logic 404 includes logic for compressing audioas well as visual material.

In operation, while playing a video game or using an application at theuser premises 211 via a keyboard, mouse, game controller or other inputdevice 421, control signal logic 413 on the client 415 transmits controlsignals 406 a-b (typically in the form of UDP packets) representing thebutton presses (and other types of user inputs) actuated by the user tothe hosting service 210. The control signals from a given user arerouted to the appropriate server (or servers, if multiple servers areresponsive to the user's input device) 402. As illustrated in FIG. 4 a ,control signals 406 a may be routed to the servers 402 via the SAN.Alternatively or in addition, control signals 406 b may be routeddirectly to the servers 402 over the hosting service network (e.g., anEthernet-based local area network). Regardless of how they aretransmitted, the server or servers execute the game or applicationsoftware in response to the control signals 406 a-b. Although notillustrated in FIG. 4 a , various networking components such as afirewall(s) and/or gateway(s) may process incoming and outgoing trafficat the edge of the hosting service 210 (e.g., between the hostingservice 210 and the Internet 410) and/or at the edge of the userpremises 211 between the Internet 410 and the home or office client 415.The graphical and audio output of the executed game or applicationsoftware—i.e., new sequences of video images—are provided to thelow-latency video compression logic 404 which compresses the sequencesof video images according to low-latency video compression techniques,such as those described herein and transmits a compressed video stream,typically with compressed or uncompressed audio, back to the client 415over the Internet 410 (or, as described below, over an optimized highspeed network service that bypasses the general Internet). Low-latencyvideo decompression logic 412 on the client 415 then decompresses thevideo and audio streams and renders the decompressed video stream, andtypically plays the decompressed audio stream, on a display device 422Alternatively, the audio can be played on speakers separate from thedisplay device 422 or not at all. Note that, despite the fact that inputdevice 421 and display device 422 are shown as free-standing devices inFIGS. 2 a and 2 b , they may be integrated within client devices such asportable computers or mobile devices.

Home or office client 415 (described previously as home or office client205 in FIGS. 2 a and 2 b ) may be a very inexpensive and low-powerdevice, with very limited computing or graphics performance and may wellhave very limited or no local mass storage. In contrast, each server402, coupled to a SAN 403 and multiple RAIDs 405 can be an exceptionallyhigh performance computing system, and indeed, if multiple servers areused cooperatively in a parallel-processing configuration, there isalmost no limit to the amount of computing and graphics processing powerthat can be brought to bear. And, because of the low-latency videocompression 404 and low-latency video compression 412, perceptually tothe user, the computing power of the servers 402 is being provided tothe user. When the user presses a button on input device 421, the imageon display 422 is updated in response to the button press perceptuallywith no meaningful delay, as if the game or application software wererunning locally. Thus, with a home or office client 415 that is a verylow performance computer or just an inexpensive chip that implements thelow-latency video decompression and control signal logic 413, a user isprovided with effectively arbitrary computing power from a remotelocation that appears to be available locally. This gives users thepower to play the most advanced, processor-intensive (typically new)video games and the highest performance applications.

FIG. 4 c shows a very basic and inexpensive home or office client device465. This device is an embodiment of home or office client 415 fromFIGS. 4 a and 4 b . It is approximately 2 inches long. It has anEthernet jack 462 that interfaces with an Ethernet cable with Power overEthernet (PoE), from which it derives its power and its connectivity tothe Internet. It is able to run Network Address Translation (NAT) withina network that supports NAT. In an office environment, many new Ethernetswitches have PoE and bring PoE directly to a Ethernet jack in anoffice. It such a situation, all that is required is an Ethernet cablefrom the wall jack to the client 465. If the available Ethernetconnection does not carry power (e.g., in a home with a DSL or cablemodem, but no PoE), then there are inexpensive wall “bricks” (i.e.,power supplies) available that will accept an unpowered Ethernet cableand output Ethernet with PoE.

The client 465 contains control signal logic 413 (of FIG. 4 a ) that iscoupled to a Bluetooth wireless interface, which interfaces withBluetooth input devices 479, such as a keyboard, mouse, game controllerand/or microphone and/or headset. Also, one embodiment of client 465 iscapable of outputting video at 120 fps coupled with a display device 468able to support 120 fps video and signal (typically through infrared) apair of shuttered glasses 466 to alternately shutter one eye, then theother with each successive frame. The effect perceived by the user isthat of a stereoscopic 3D image that “jumps out” of the display screen.One such display device 468 that supports such operation is the SamsungHL-T5076S. Since the video stream for each eye is separate, in oneembodiment two independent video streams are compressed by the hostingservice 210, the frames are interleaved in time, and the frames aredecompressed as two independent decompression processes within client465.

The client 465 also contains low latency video decompression logic 412,which decompresses the incoming video and audio and output through theHDMI (High-Definition Multimedia Interface), connector 463 which plugsinto an SDTV (Standard Definition Television) or HDTV (High DefinitionTelevision) 468, providing the TV with video and audio, or into amonitor 468 that supports HDMI. If the user's monitor 468 does notsupport HDMI, then an HDMI-to-DVI (Digital Visual Interface) can beused, but the audio will be lost. Under the HDMI standard, the displaycapabilities (e.g. supported resolutions, frame rates) 464 arecommunicated from the display device 468, and this information is thenpassed back through the Internet connection 462 back to the hostingservice 210 so it can stream compressed video in a format suitable forthe display device.

FIG. 4 d shows a home or office client device 475 that is the same asthe home or office client device 465 shown in FIG. 4 c except that ishas more external interfaces. Also, client 475 can accept either PoE forpower, or it can run off of an external power supply adapter (not shown)that plugs in the wall. Using client 475 USB input, video camera 477provides compressed video to client 475, which is uploaded by client 475to hosting service 210 for use described below. Built into camera 477 isa low-latency compressor utilizing the compression techniques describedbelow.

In addition to having an Ethernet connector for its Internet connection,client 475 also has an 802.11g wireless interface to the Internet. Bothinterfaces are able to use NAT within a network that supports NAT.

Also, in addition to having an HDMI connector to output video and audio,client 475 also has a Dual Link DVI-I connector, which includes analogoutput (and with a standard adapter cable will provide VGA output). Italso has analog outputs for composite video and S-video.

For audio, the client 475 has left/right analog stereo RCA jacks, andfor digital audio output it has a TOSLINK output.

In addition to a Bluetooth wireless interface to input devices 479, italso has USB jacks to interface to input devices.

FIG. 4 e shows one embodiment of the internal architecture of client465. Either all or some of the devices shown in the diagram can beimplemented in an Field Programmable Logic Array, an custom ASIC or inseveral discrete devices, either custom designed or off-the-shelf.

Ethernet with PoE 497 attaches to Ethernet Interface 481. Power 499 isderived from the Ethernet with PoE 497 and is connected to the rest ofthe devices in the client 465. Bus 480 is a common bus for communicationbetween devices.

Control CPU 483 (almost any small CPU, such as a MIPS R4000 series CPUat 100 MHz with embedded RAM is adequate) running a small client controlapplication from Flash 476 implements the protocol stack for the network(i.e. Ethernet interface) and also communicates with the Hosting Service210, and configures all of the devices in the client 465. It alsohandles interfaces with the input devices 469 and sends packets back tothe hosting service 210 with user controller data, protected by ForwardError Correction, if necessary. Also, Control CPU 483 monitors thepacket traffic (e.g. if packets are lost or delayed and also timestampstheir arrival). This information is sent back to the hosting service 210so that it can constantly monitor the network connection and adjust whatit sends accordingly. Flash memory 476 is initially loaded at the timeof manufacture with the control program for Control CPU 483 and alsowith a serial number that is unique to the particular Client 465 unit.This serial number allows the hosting service 210 to uniquely identifythe Client 465 unit.

Bluetooth interface 484 communicates to input devices 469 wirelesslythrough its antenna, internal to client 465.

Video decompressor 486 is a low-latency video decompressor configured toimplement the video decompression described herein. A large number ofvideo decompression devices exist, either off-the-shelf, or asIntellectual Property (IP) of a design that can be integrated into anFPGA or a custom ASIC. One company offering IP for an H.264 decoder isOcean Logic of Manly, NSW Australia. The advantage of using IP is thatthe compression techniques used herein do not conform to compressionstandards. Some standard decompressors are flexible enough to beconfigured to accommodate the compression techniques herein, but somecan not. But, with IP, there is complete flexibility in redesigning thedecompressor as needed.

The output of the video decompressor is coupled to the video outputsubsystem 487, which couples the video to the video output of the HDMIinterface 490.

The audio decompression subsystem 488 is implemented either using astandard audio decompressor that is available, or it can be implementedas IP, or the audio decompression can be implemented within the controlprocessor 483 which could, for example, implement the Vorbis audiodecompressor.

The device that implements the audio decompression is coupled to theaudio output subsystem 489 that couples the audio to the audio output ofthe HDMI interface 490

FIG. 4 f shows one embodiment of the internal architecture of client475. As can be seen, the architecture is the same as that of client 465except for additional interfaces and optional external DC power from apower supply adapter that plugs in the wall, and if so used, replacespower that would come from the Ethernet PoE 497. The functionality thatis in common with client 465 will not be repeated below, but theadditional functionality is described as follows.

CPU 483 communicates with and configures the additional devices.

WiFi subsystem 482 provides wireless Internet access as an alternativeto Ethernet 497 through its antenna. WiFi subsystems are available froma wide range of manufacturers, including Atheros Communications of SantaClara, Calif.

USB subsystem 485 provides an alternative to Bluetooth communication forwired USB input devices 479. USB subsystems are quite standard andreadily available for FPGAs and ASICs, as well as frequently built intooff-the-shelf devices performing other functions, like videodecompression.

Video output subsystem 487 produces a wider range of video outputs thanwithin client 465. In addition to providing HDMI 490 video output, itprovides DVI-I 491, S-video 492, and composite video 493. Also, when theDVI-I 491 interface is used for digital video, display capabilities 464are passed back from the display device to the control CPU 483 so thatit can notify the hosting service 210 of the display device 478capabilities. All of the interfaces provided by the video outputsubsystem 487 are quite standard interfaces and readily available inmany forms.

Audio output subsystem 489 outputs audio digitally through digitalinterface 494 (S/PDIF and/or TOSLINK) and audio in analog form throughstereo analog interface 495.

Round-Trip Latency Analysis

Of course, for the benefits of the preceding paragraph to be realized,the round trip latency between a user's action using input device 421and seeing the consequence of that action on display device 420 shouldbe no more than 70-80 ms. This latency must take into account all of thefactors in the path from input device 421 in the user premises 211 tohosting service 210 and back again to the user premises 211 to displaydevice 422. FIG. 4 b illustrates the various components and networksover which signals must travel, and above these components and networksis a timeline that lists exemplary latencies that can be expected in apractical implementation. Note that FIG. 4 b is simplified so that onlythe critical path routing is shown. Other routing of data used for otherfeatures of the system is described below. Double-headed arrows (e.g.,arrow 453) indicate round-trip latency and a single-headed arrow (e.g.,arrow 457) indicate one-way latency, and “˜” denote an approximatemeasure. It should be pointed out that there will be real-worldsituations where the latencies listed can not be achieved, but in alarge number of cases in the US, using DSL and cable modem connectionsto the user premises 211, these latencies can be achieved in thecircumstances described in the next paragraph. Also, note that, whilecellular wireless connectivity to the Internet will certainly work inthe system shown, most current US cellular data systems (such as EVDO)incur very high latencies and would not be able to achieve the latenciesshown in FIG. 4 b . However, these underlying principles may beimplemented on future cellular technologies that may be capable ofimplementing this level of latency.

Starting from the input device 421 at user premises 211, once the useractuates the input device 421, a user control signal is sent to client415 (which may be a standalone device such a set-top box, or it may besoftware or hardware running in another device such as a PC or a mobiledevice), and is packetized (in UDP format in one embodiment) and thepacket is given a destination address to reach hosting service 210. Thepacket will also contain information to indicate which user the controlsignals are coming from. The control signal packet(s) are then forwardedthrough Firewall/Router/NAT (Network Address Translation) device 443 toWAN interface 442. WAN interface 442 is the interface device provided tothe user premises 211 by the User's ISP (Internet Service Provider). TheWAN interface 442 may be a Cable or DSL modem, a WiMax transceiver, aFiber transceiver, a Cellular data interface, a InternetProtocol-over-powerline interface, or any other of many interfaces tothe Internet. Further, Firewall/Router/NAT device 443 (and potentiallyWAN interface 442) may be integrated into the client 415. An example ofthis would be a mobile phone, which includes software to implement thefunctionality of home or office client 415, as well as the means toroute and connect to the Internet wirelessly through some standard(e.g., 802.11g).

WAN Interface 442 then routes the control signals to what shall becalled herein the “point of presence” 441 for the user's InternetService Provider (ISP) which is the facility that provides an interfacebetween the WAN transport connected to the user premises 211 and thegeneral Internet or private networks. The point of presence'scharacteristics will vary depending upon nature of the Internet serviceprovided. For DSL, it typically will be a telephone company CentralOffice where a DSLAM is located. For cable modems, it typically will bea cable Multi-System Operator (MSO) head end. For cellular systems, ittypically will be a control room associated with cellular tower. Butwhatever the point of presence's nature, it will then route the controlsignal packet(s) to the general Internet 410. The control signalpacket(s) will then be routed to the WAN Interface 441 to the hostingservice 210, through what most likely will be a fiber transceiverinterface. The WAN 441 will then route the control signal packets torouting logic 409 (which may be implemented in many different ways,including Ethernet switches and routing servers), which evaluates theuser's address and routes the control signal(s) to the correct server402 for the given user.

The server 402 then takes the control signals as input for the game orapplication software that is running on the server 402 and uses thecontrol signals to process the next frame of the game or application.Once the next frame is generated, the video and audio is output fromserver 402 to video compressor 404. The video and audio may be outputfrom server 402 to compressor 404 through various means. To start with,compressor 404 may be built into server 402, so the compression may beimplemented locally within server 402. Or, the video and/or audio may beoutput in packetized form through a network connection such as anEthernet connection to a network that is either a private networkbetween server 402 and video compressor 404, or a through a sharednetwork, such as SAN 403. Or, the video may be output through a videooutput connector from server 402, such as a DVI or VGA connector, andthen captured by video compressor 404. Also, the audio may be outputfrom server 402 as either digital audio (e.g., through a TOSLINK orS/PDIF connector) or as analog audio, which is digitized and encoded byaudio compression logic within video compressor 404.

Once video compressor 404 has captured the video frame and the audiogenerated during that frame time from server 402, then video compressorwill compress the video and audio using techniques described below. Oncethe video and audio is compressed it is packetized with an address tosend it back to the user's client 415, and it is routed to the WANInterface 441, which then routes the video and audio packets through thegeneral Internet 410, which then routes the video and audio packets tothe user's ISP point of presence 441, which routes the video and audiopackets to the WAN Interface 442 at the user's premises, which routesthe video and audio packets to the Firewall/Router/NAT device 443, whichthen routes the video and audio packets to the client 415.

The client 415 decompresses the video and audio, and then displays thevideo on the display device 422 (or the client's built-in displaydevice) and sends the audio to the display device 422 or to separateamplifier/speakers or to an amplifier/speakers built in the client.

For the user to perceive that the entire process just described isperceptually without lag, the round-trip delay needs be less than 70 or80 ms. Some of the latency delays in the described round-trip path areunder the control of the hosting service 210 and/or the user and othersare not. Nonetheless, based on analysis and testing of a large number ofreal-world scenarios, the following are approximate measurements.

The one-way transmission time to send the control signals 451 istypically less than 1 ms, the roundtrip routing through the userpremises 452 is typically accomplished, using readily availableconsumer-grade Firewall/Router/NAT switches over Ethernet in about 1 ms.User ISPs vary widely in their round trip delays 453, but with DSL andcable modem providers, we typically see between 10 and 25 ms. The roundtrip latency on the general Internet 410 can vary greatly depending onhow traffic is routed and whether there are any failures on the route(and these issues are discussed below), but typically the generalInternet provides fairly optimal routes and the latency is largelydetermined by speed of light through optical fiber, given the distanceto the destination. As discussed further below, we have established 1000miles as a roughly the furthest distance that we expect to place ahosting service 210 away from user premises 211. At 1000 miles (2000miles round trip) the practical transit time for a signal through theInternet is approximately 22 ms. The WAN Interface 441 to the hostingservice 210 is typically a commercial-grade fiber high speed interfacewith negligible latency. Thus, the general Internet latency 454 istypically between 1 and 10 ms. The one-way routing 455 latency throughthe hosting service 210 can be achieved in less than 1 ms. The server402 will typically compute a new frame for a game or an application inless than one frame time (which at 60 fps is 16.7 ms) so 16 ms is areasonable maximum one-way latency 456 to use. In an optimized hardwareimplementation of the video compression and audio compression algorithmsdescribed herein, the compression 457 can be completed in 1 ms. In lessoptimized versions, the compression may take as much as 6 ms (of courseeven less optimized versions could take longer, but such implementationswould impact the overall latency of the round trip and would requireother latencies to be shorter (e.g., the allowable distance through thegeneral Internet could be reduced) to maintain the 70-80 ms latencytarget). The round trip latencies of the Internet 454, User ISP 453, andUser Premises Routing 452 have already been considered, so what remainsis the video decompression 458 latency which, depending on whether thevideo decompression 458 is implemented in dedicated hardware, or ifimplemented in software on a client device 415 (such as a PC or mobiledevice) it can vary depending upon the size of the display and theperformance of the decompressing CPU. Typically, decompression 458 takesbetween 1 and 8 ms.

Thus, by adding together all of the worst-case latencies seen inpractice, we can determine the worst-case round trip latency that can beexpected to be experience by a user of the system shown in FIG. 4 a .They are: 1+1+25+22+1+16+6+8=80 ms. And, indeed, in practice (withcaveats discussed below), this is roughly the round trip latency seenusing prototype versions of the system shown in FIG. 4 a , usingoff-the-shelf Windows PCs as client devices and home DSL and cable modemconnections within the US. Of course, scenarios better than worst casecan result in much shorter latencies, but they can not be relied upon indeveloping a commercial service that is used widely.

To achieve the latencies listed in FIG. 4 b over the general Internet,requires the video compressor 404 and video decompressor 412 from FIG. 4a in the client 415 to generate a packet stream which very particularcharacteristics, such that the packet sequence generated through entirepath from the hosting service 210 to the display device 422 is notsubject to delays or excessive packet loss and, in particular,consistently falls with the constraints of the bandwidth available tothe user over the user's Internet connection through WAN interface 442and Firewall/Router/NAT 443. Further, the video compressor must create apacket stream which is sufficiently robust so that it can tolerate theinevitable packet loss and packet reordering that occurs in normalInternet and network transmissions.

Low-Latency Video Compression

To accomplish the foregoing goals, one embodiment takes a new approachto video compression which decreases the latency and the peak bandwidthrequirements for transmitting video. Prior to the description of theseembodiments, an analysis of current video compression techniques will beprovided with respect to FIG. 5 and FIGS. 6 a-b . Of course, thesetechniques may be employed in accordance with underlying principles ifthe user is provided with sufficient bandwidth to handle the data raterequired by these techniques. Note that audio compression is notaddressed herein other than to state that it is implementedsimultaneously and in synchrony with the video compression. Prior artaudio compression techniques exist that satisfy the requirements forthis system.

FIG. 5 illustrates one particular prior art technique for compressingvideo in which each individual video frame 501-503 is compressed bycompression logic 520 using a particular compression algorithm togenerate a series of compressed frames 511-513. One embodiment of thistechnique is “motion JPEG” in which each frame is compressed accordingto a Joint Pictures Expert Group (JPEG) compression algorithm, basedupon the discrete cosine transform (DCT). Various different types ofcompression algorithms may be employed, however, while still complyingwith these underlying principles (e.g., wavelet-based compressionalgorithms such as JPEG-2000).

One problem with this type of compression is that it reduces the datarate of each frame, but it does not exploit similarities betweensuccessive frames to reduce the data rate of the overall video stream.For example, as illustrated in FIG. 5 , assuming a frame rate of640×480×24 bits/pixel=640*480*24/8/1024=900 Kilobytes/frame (KB/frame),for a given quality of image, motion JPEG may only compress the streamby a factor of 10, resulting in a data stream of 90 KB/frame. At 60frames/sec, this would require a channel bandwidth of 90 KB*8 bits*60frames/sec=42.2 Mbps, which would be far too high bandwidth for almostall home Internet connections in the US today, and too high bandwidthfor many office Internet connections. Indeed, given that it would demanda constant data stream at such a high bandwidth, and it would be justserving one user, even in an office LAN environment, it would consume alarge percentage of a 100 Mbps Ethernet LAN's bandwidth and heavilyburden Ethernet switches supporting the LAN. Thus, the compression formotion video is inefficient when compared with other compressiontechniques (such as those described below). Moreover, single framecompression algorithms like JPEG and JPEG-2000 that use lossycompression algorithms produce compression artifacts that may not benoticeable in still images (e.g., an artifact within dense foliage inthe scene may not appear as an artifact since the eye does not knowexactly how the dense foliage should appear). But, once the scene is inmotion, an artifact can stand out because the eye detects that theartifact changed from frame-to-frame, despite the fact the artifact isin an area of the scene where it might not have been noticeable in astill image. This results in the perception of “background noise” in thesequence of frames, similar in appearance to the “snow” noise visibleduring marginal analog TV reception. Of course, this type of compressionmay still be used in certain embodiments described herein, but generallyspeaking, to avoid background noise in the scene, a high data rate(i.e., a low compression ratio) is required for a given perceptualquality.

Other types of compression, such as H.264, or Windows Media VC9, MPEG2and MPEG4 are all more efficient at compressing a video stream becausethey exploit the similarities between successive frames. Thesetechniques all rely upon the same general techniques to compress video.Thus, although the H.264 standard will be described, but the samegeneral principles apply to various other compression algorithms. Alarge number of H.264 compressors and decompressor are available,including the x264 open source software library for compressing H.264and the FFmpeg open source software libraries for decompressing H.264.

FIGS. 6 a and 6 b illustrate an exemplary prior art compressiontechnique in which a series of uncompressed video frames 501-503,559-561 are compressed by compression logic 620 into a series of “Iframes” 611, 671; “P frames” 612-613; and “B frames” 670. The verticalaxis in FIG. 6 a generally signifies the resulting size of each of theencoded frames (although the frames are not drawn to scale). Asdescribed above, video coding using I frames, B frames and P frames iswell understood by those of skill in the art. Briefly, an I frame 611 isa DCT-based compression of a complete uncompressed frame 501 (similar toa compressed JPEG image as described above). P frames 612-613 generallyare significantly smaller in size than I frames 611 because they takeadvantage of the data in the previous I frame or P frame; that is, theycontain data indicating the changes between the previous I frame or Pframe. B frames 670 are similar to that of P frames except that B framesuse the frame in the following reference frame as well as potentiallythe frame in the preceding reference frame.

For the following discussion, it will be assumed that the desired framerate is 60 frames/second, that each I frame is approximately 160 Kb, theaverage P frame and B frame is 16 Kb and that a new I frame is generatedevery second. With this set of parameters, the average data rate wouldbe: 160 Kb+16 Kb*59=1.1 Mbps. This data rate falls well within themaximum data rate for many current broadband Internet connections tohomes and offices. This technique also tends to avoid the backgroundnoise problem from intraframe-only encoding because the P and B framestrack differences between the frames, so compression artifacts tend notto appear and disappear from frame-to-frame, thereby reducing thebackground noise problem described above.

One problem with the foregoing types of compression is that although theaverage data rate is relatively low (e.g., 1.1 Mbps), a single I framemay take several frame times to transmit. For example, using prior arttechniques a 2.2 Mbps network connection (e.g., DSL or cable modem with2.2 Mbps peak of max available data rate 302 from FIG. 3 a ) wouldtypically be adequate to stream video at 1.1 Mbps with a 160 Kbps Iframe each 60 frames. This would be accomplished by having thedecompressor queue up 1 second of video before decompressing the video.In 1 second, 1.1 Mb of data would be transmitted, which would be easilyaccommodated by a 2.2 Mbps max available data rate, even assuming thatthe available data rate might dip periodically by as much as 50%.Unfortunately, this prior art approach would result in a 1-secondlatency for the video because of the 1-second video buffer at thereceiver. Such a delay is adequate for many prior art applications(e.g., the playback of linear video), but is far too long a latency forfast action video games which cannot tolerate more than 70-80 ms oflatency.

If an attempt were made to eliminate the 1-second video buffer, it stillwould not result in an adequate reduction in latency for fast actionvideo games. For one, the use of B frames, as previously described,would necessitate the reception of all of the B frames preceding an Iframe as well as the I frame. If we assume the 59 non-I frames areroughly split between P and B frames, then there would be at least 29 Bframes and an I frame received before any B frame could be displayed.Thus, regardless of the available bandwidth of the channel, it wouldnecessitate a delay of 29+1=30 frames of 1/60^(th) second duration each,or 500 ms of latency. Clearly that is far too long.

Thus, another approach would be to eliminate B frames and only use I andP frames. (One consequence of this is the data rate would increase for agiven quality level, but for the sake of consistency in this example,let's continue to assume that each I frame is 160 Kb and the average Pframe is 16 Kb in size, and thus the data rate is still 1.1 Mbps) Thisapproach eliminates the unavoidable latency introduced by B frames,since the decoding of each P frame is only reliant upon the priorreceived frame. A problem that remains with this approach is that an Iframe is so much larger than an average P frame, that on a low bandwidthchannel, as is typical in most homes and in many offices, thetransmission of the I frame adds substantial latency. This isillustrated in FIG. 6 b . The video stream data rate 624 is below theavailable max data rate 621 except for the I frames, where the peak datarate required for the I frames 623 far exceeds the available max datarate 622 (and even the rated max data rate 621). The data rate requiredby the P frames is less than the available max data rate. Even if theavailable max data rate peaks at 2.2 Mbps remains steadily at its 2.2Mbps peak rate, it will take 160 Kb/2.2 Mb=71 ms to transmit the Iframe, and if the available max data rate 622 dips by 50% (1.1 Mbps), itwill take 142 ms to transmit the I frame. So, the latency intransmitting the I frame will fall somewhere in between 71-142 ms. Thislatency is additive to the latencies identified in FIG. 4 b , which inthe worst case added up to 70 ms, so this would result in a total roundtrip latency of 141-222 ms from the point the user actuates input device421 until an image appears on display device 422, which is far too high.And if the available max data rate dips below 2.2 Mbps, the latency willincrease further.

Note also that there generally are severe consequences to “jamming” anISP with peak data rate 623 that are far in excess of the available datarate 622. The equipment in different ISPs will behave differently, butthe following behaviors are quite common among DSL and cable modem ISPswhen receiving packets at much higher data rate than the available datarate 622: (a) delaying the packets by queuing them (introducinglatency), (b) dropping some or all of the packets, (c) disabling theconnection for a period of time (most likely because the ISP isconcerned it is a malicious attack, such as “denial of service” attack).Thus, transmitting a packet stream at full data rate withcharacteristics such as those shown in FIG. 6 b is not a viable option.The peaks 623 may be queued up at the hosting service 210 and sent at adata rate below the available maximum data rate, introducing theunacceptable latency described in the preceding paragraph.

Further, the video stream data rate sequence 624 shown in FIG. 6 b is avery “tame” video stream data rate sequence and would be the sort ofdata rate sequence that one would expect to result from compressing thevideo from a video sequence that does not change very much and has verylittle motion (e.g., as would be common in video teleconferencing wherethe cameras are in a fixed position and have little motion, and theobjects, in the scene, e.g., seated people talking, show little motion).

The video stream data rate sequence 634 shown in FIG. 6 c is a sequencetypical to what one would expect to see from video with far more action,such as might be generated in a motion picture or a video game, or insome application software. Note that in addition to the I frame peaks633, there are also P frame peaks such as 635 and 636 that are quitelarge and exceed the available max data rate on many occasions. Althoughthese P frame peaks are not quite as large as the I frame peaks, theystill are far too large to be carried by the channel at full data rate,and as with the I frame peaks, they P frame peaks must be transmittedslowly (thereby increasingly latency).

On a high bandwidth channel (e.g., a 100 Mbps LAN, or a high bandwidth100 Mbps private connection) the network would be able to tolerate largepeaks, such as I frame peaks 633 or P frame peaks 636, and in principle,low latency could be maintained. But, such networks are frequentlyshared amongst many users (e.g., in an office environment), and such“peaky” data would impact the performance of the LAN, particularly ifthe network traffic was routed to a private shared connection (e.g.,from a remote data center to an office). To start with, bear in mindthat this example is of a relatively low resolution video stream of640×480 pixels at 60 fps. HDTV streams of 1920×1080 at 60 fps arereadily handled by modern computers and displays, and 2560×1440resolution displays at 60 fps are increasingly available (e.g., Apple,Inc.'s 30″ display). A high action video sequence at 1920×1080 at 60 fpsmay require 4.5 Mbps using H.264 compression for a reasonable qualitylevel. If we assume the I frames peak at 10× the nominal data rate, thatwould result in 45 Mbps peaks, as well as smaller, but stillconsiderable, P frame peak. If several users were receiving videostreams on the same 100 Mbps network (e.g., a private network connectionbetween an office and data center), it is easy to see how the peaks fromseveral users' video stream could happen to align, overwhelming thebandwidth of the network, and potentially overwhelming the bandwidth ofthe backplanes of the switches supporting the users on the network. Evenin the case of a Gigabit Ethernet network, if enough users had enoughpeaks aligned at once, it could overwhelm the network or the networkswitches. And, once 2560×1440 resolution video becomes more commonplace,the average video stream data rate may be 9.5 Mbps, resulting in perhapsa 95 Mbps peak data rate. Needless to say, a 100 Mbps connection betweena data center and an office (which today is an exceptionally fastconnection) would be completely swamped by the peak traffic from asingle user. Thus, even though LANs and private network connections canbe more tolerant of peaky streaming video, the streaming video with highpeaks is not desirable and might require special planning andaccommodation by an office's IT department.

Of course, for standard linear video applications these issues are not aproblem because the data rate is “smoothed” at the point of transmissionand the data for each frame below the max available data rate 622, and abuffer in the client stores a sequence of I, P and B frames before theyare decompressed. Thus, the data rate over the network remains close tothe average data rate of the video stream. Unfortunately, thisintroduces latency, even if B frames are not used, that is unacceptablefor low-latency applications such as video games and applicationsrequire fast response time.

One prior art solution to mitigating video streams that have high peaksis to use a technique often referred to as “Constant Bit Rate” (CBR)encoding. Although the term CBR would seem to imply that all frames arecompressed to have the same bit rate (i.e., size), what it usuallyrefers to is a compression paradigm where a maximum bit rate across acertain number of frames (in our case, 1 frame) is allowed. For example,in the case of FIG. 6 c , if a CBR constraint were applied to theencoding that limited the bit rate to, for example, 70% of the rated maxdata rate 621, then the compression algorithm would limit thecompression of each of the frames so that any frame that would normallybe compressed using more than 70% of the rated max data rate 621 wouldbe compressed with less bits. The result of this is that frames thatwould normally require more bits to maintain a given quality level wouldbe “starved” of bits and the image quality of those frames would beworse than that of other frames that do not require more bits than the70% of the rate max data rate 621. This approach can produce acceptableresults for certain types of compressed video where there (a) littlemotion or scene changes are expected and (b) the users can acceptperiodic quality degradation. A good example of a CBR-suited applicationis video teleconferencing since there are few peaks, and if the qualitydegrades briefly (for example, if the camera is panned, resulting insignificant scene motion and large peaks, during the panning there maynot be enough bits for high-quality image compression, which couldresult in degraded image quality), it is acceptable for most users.Unfortunately, CBR is not well-suited for many other applications whichhave scenes of high complexity or a great deal of motion and/or where areasonably constant level of quality is required.

The low-latency compression logic 404 employed in one embodiment usesseveral different techniques to address the range of problems withstreaming low-latency compressed video, while maintaining high quality.First, the low-latency compression logic 404 generates only I frames andP frames, thereby alleviating the need to wait several frame times todecode each B frame. In addition, as illustrated in FIG. 7 a , in oneembodiment, the low-latency compression logic 404 subdivides eachuncompressed frame 701-760 into a series of “tiles” and individuallyencodes each tile as either an I frame or a P frame. The group ofcompressed I frames and P frames are referred to herein as “R frames”711-770. In the specific example shown in FIG. 7 a , each uncompressedframe is subdivided into a 4×4 matrix of 16 tiles. However, theseunderlying principles are not limited to any particular subdivisionscheme.

In one embodiment, the low-latency compression logic 404 divides up thevideo frame into a number of tiles, and encodes (i.e., compresses) onetile from each frame as an I frame (i.e., the tile is compressed as ifit is a separate video frame of 1/16^(th) the size of the full image,and the compression used for this “mini” frame is I frame compression)and the remaining tiles as P frames (i.e., the compression used for each“mini” 1/16^(th) frame is P frame compression). Tiles compressed as Iframes and as P frames shall be referred to as “I tiles” and “P tiles”,respectively. With each successive video frame, the tile to be encodedas an I tile is changed. Thus, in a given frame time, only one tile ofthe tiles in the video frame is an I tile, and the remainder of thetiles are P tiles. For example, in FIG. 7 a , tile 0 of uncompressedframe 701 is encoded as I tile I₀ and the remaining 1-15 tiles areencoded as P tiles P₁ through P₁₅ to produce R frame 711. In the nextuncompressed video frame 702, tile 1 of uncompressed frame 701 isencoded as I tile I₁ and the remaining tiles 0 and 2 through 15 areencoded as P tiles, P₀ and P₂ through P₁₅, to produce R frame 712. Thus,the I tiles and P tiles for tiles are progressively interleaved in timeover successive frames. The process continues until an R tile 770 isgenerated with the last tile in the matrix encoded as an I tile (i.e.,I₁₅). The process then starts over, generating another R frame such asframe 711 (i.e., encoding an I tile for tile 0) etc. Although notillustrated in FIG. 7 a , in one embodiment, the first R frame of thevideo sequence of R frames contains only I tiles (i.e., so thatsubsequent P frames have reference image data from which to calculatemotion). Alternatively, in one embodiment, the startup sequence uses thesame I tile pattern as normal, but does not include P tiles for thosetiles that have not yet been encoded with an I tile. In other words,certain tiles are not encoded with any data until the first I tilearrives, thereby avoiding startup peaks in the video stream data rate934 in FIG. 9 a , which is explained in further detail below. Moreover,as described below, various different sizes and shapes may be used forthe tiles while still complying with these underlying principles.

The video decompression logic 412 running on the client 415 decompresseseach tile as if it is a separate video sequence of small I and P frames,and then renders each tile to the frame buffer driving display device422. For example, I₀ and P₀ from R frames 711 to 770 are used todecompress and render tile 0 of the video image. Similarly, I₁ and P₁from R frames 711 to 770 are used to reconstruct tile 1, and so on. Asmentioned above, decompression of I frames and P frames is well known inthe art, and decompression of I tiles and P tiles can be accomplished byhaving a multiple instances of a video decompressor running in theclient 415. Although multiplying processes would seem to increase thecomputational burden on client 415, it actually does not because thetile themselves are proportionally smaller relative to the number ofadditional processes, so the number of pixels displayed is the same asif there were one process and using conventional full sized I and Pframes.

This R frame technique significantly mitigates the bandwidth peakstypically associated with I frames illustrated in FIGS. 6 b and 6 cbecause any given frame is mostly made up of P frames which aretypically smaller than I frames. For example, assuming again that atypical I frame is 160 Kb, then the I tiles of each of the framesillustrated in FIG. 7 a would be roughly 1/16 of this amount or 10 Kb.Similarly, assuming that a typical P frame is 16 Kb, then the P framesfor each of the tiles illustrated in FIG. 7 a may be roughly 1 Kb Theend result is an R frame of approximately 10 Kb+15*1 Kb=25 Kb. So, each60-frame sequence would be 25 Kb*60=1.5 Mbps. So, at 60 frames/second,this would require a channel capable of sustaining a bandwidth of 1.5Mbps, but with much lower peaks due to I tiles being distributedthroughout the 60-frame interval.

Note that in previous examples with the same assumed data rates for Iframes and P frames, the average data rate was 1.1 Mbps. This is becausein the previous examples, a new I frame was only introduced once every60 frame times, whereas in this example, the 16 tiles that make up an Iframe cycle through in 16 frames times, and as such the equivalent of anI frame is introduced every 16 frame times, resulting in a slightlyhigher average data rate. In practice, though, introducing more frequentI frames does not increase the data rate linearly. This is due to thefact that a P frame (or a P tile) primarily encodes the difference fromthe prior frame to the next. So, if the prior frame is quite similar tothe next frame, the P frame will be very small, if the prior frame isquite different from the next frame, the P frame will be very large. Butbecause a P frame is largely derived from the previous frame, ratherthan from the actual frame, the resulting encoded frame may contain moreerrors (e.g., visual artifacts) than an I frame with an adequate numberof bits. And, when one P frame follows another P frame, what can occuris an accumulation of errors that gets worse when there is a longsequence of P frames. Now, a sophisticated video compressor will detectthe fact that the quality of the image is degrading after a sequence ofP frames and, if necessary, it will allocate more bits to subsequent Pframes to bring up the quality or, if it is the most efficient course ofaction, replace a P frame with an I frame. So, when long sequences of Pframes are used (e.g., 59 P frames, as in prior examples above)particularly when the scene has a great deal of complexity and/ormotion, typically, more bits are needed for P frames as they get furtherremoved from an I frame.

Or, to look at P frames from the opposite point of view, P frames thatclosely follow an I frame tend to require less bits than P frames thatare further removed from an I frame. So, in the example shown in FIG. 7a , no P frame is further than 15 frames removed from an I frame thatprecedes it, where as in the prior example, a P frame could be 59 framesremoved from an I frame. Thus, with more frequent I frames, the P framesare smaller. Of course, the exact relative sizes will vary based on thenature of the video stream, but in the example of FIG. 7 a , if an Itile is 10 Kb, P tiles on average, may be only 0.75 kb in size resultingin 10 Kb+15*0.75 Kb=21.25 Kb, or at 60 frames per second, the data ratewould be 21.25 Kb*60=1.3 Mbps, or about 16% higher data rate than astream with an I frame followed by 59 P frames at 1.1 Mbps. Once, again,the relative results between these two approaches to video compressionwill vary depending up on the video sequence, but typically, we havefound empirically that using R-frames require about 20% more bits for agiven level of quality than using I/P frame sequences. But, of course, Rframes dramatically reduce the peaks which make the video sequencesusable with far less latency than I/P frame sequences.

R frames can be configured in a variety of different ways, dependingupon the nature of the video sequence, the reliability of the channel,and the available data rate. In an alternative embodiment, a differentnumber of tiles is used than 16 in a 4×4 configuration. For example 2tiles may be used in a 2×1 or 1×2 configuration, 4 tiles may be used ina 2×2, 4×1 or 1×4 configuration, 6 tiles may be used in a 3×2, 2×3, 6×1or 1×6 configurations or 8 tiles may be used in a 4×2 (as shown in FIG.7 b ), 2×4, 8×1 or 1×8 configuration. Note that the tiles need not besquare, nor must the video frame be square, or even rectangular. Thetiles can be broken up into whatever shape best suits the video streamand the application used.

In another embodiment, the cycling of the I and P tiles is not locked tothe number of tiles. For example, in an 8-tile 4×2 configuration, a16-cycle sequence can still be used as illustrated in FIG. 7 b .Sequential uncompressed frames 721, 722, 723 are each divided into 8tiles, 0-7 and each tile is compressed individually. From R frame 731,only tile 0 is compressed as an I tile, and the remaining tiles arecompressed as P tiles. For subsequent R frame 732 all of the 8 tiles arecompressed as P tiles, and then for subsequent R frame 733, tile 1 iscompressed as an I tile and the other tiles are all compressed as Ptiles. And, so the sequencing continues for 16 frames, with an I tilegenerated only every other frame, so the last I tile is generated fortile 7 during the 15^(th) frame time (not shown in FIG. 7 b ) and duringthe 16^(th) frame time R frame 780 is compressed using all P tiles.Then, the sequence begins again with tile 0 compressed as an I tile andthe other tiles compressed as P tiles. As in the prior embodiment, thevery first frame of the entire video sequence would typically be all Itiles, to provide a reference for P tiles from that point forward. Thecycling of I tiles and P tiles need not even be an even multiple of thenumber of tiles. For example, with 8 tiles, each frame with an I tilecan be followed by 2 frames with all P tiles, before another I tile isused. In yet another embodiment, certain tiles may be sequenced with Itiles more often than other tiles if, for example, certain areas of thescreen are known to have more motion requiring from frequent I tiles,while others are more static (e.g., showing a score for a game)requiring less frequent I tiles. Moreover, although each frame isillustrated in FIGS. 7 a-b with a single I tile, multiple I tiles may beencoded in a single frame (depending on the bandwidth of thetransmission channel). Conversely, certain frames or frame sequences maybe transmitted with no I tiles (i.e., only P tiles).

The reason the approaches of the preceding paragraph works well is thatwhile not having I tiles distributed across every single frame wouldseem to be result in larger peaks, the behavior of the system is notthat simple. Since each tile is compressed separately from the othertiles, as the tiles get smaller the encoding of each tile can becomeless efficient, because the compressor of a given tile is not able toexploit similar image features and similar motion from the other tiles.Thus, dividing up the screen into 16 tiles generally will result in aless efficient encoding than dividing up the screen into 8 tiles. But,if the screen is divided into 8 tiles and it causes the data of a full Iframe to be introduced every 8 frames instead of every 16 frames, itresults in a much higher data rate overall. So, by introducing a full Iframe every 16 frames instead of every 8 frames, the overall data rateis reduced. Also, by using 8 larger tiles instead of 16 smaller tiles,the overall data rate is reduced, which also mitigates to some degreethe data peaks caused by the larger tiles.

In another embodiment, the low-latency video compression logic 404 inFIGS. 7 a and 7 b controls the allocation of bits to the various tilesin the R frames either by being pre-configured by settings, based onknown characteristics of the video sequence to be compressed, orautomatically, based upon an ongoing analysis of the image quality ineach tile. For example, in some racing video games, the front of theplayer's car (which is relatively motionless in the scene) takes up alarge part of the lower half of the screen, whereas the upper half ofthe screen is entirely filled with the oncoming roadway, buildings andscenery, which is almost always in motion. If the compression logic 404allocates an equal number of bits to each tile, then the tiles on thebottom half of the screen (tiles 4-7) in uncompressed frame 721 in FIG.7 b , will generally be compressed with higher quality than tiles thanthe tiles in the upper half of the screen (tiles 0-3) in uncompressedframe 721 in FIG. 7 b . If this particular game, or this particularscene of the game is known to have such characteristics, then theoperators of the hosting service 210 can configure the compression logic404 to allocate more bits to the tiles in the top of the screen than totiles at the bottom of the screen. Or, the compression logic 404 canevaluate the quality of the compression of the tiles after frames arecompressed (using one or more of many compression quality metrics, suchas Peak Signal-To-Noise Ratio (PSNR)) and if it determines that over acertain window of time, certain tiles are consistently producing betterquality results, then it gradually allocates more bits to tiles that areproducing lower quality results, until the various tiles reach a similarlevel of quality. In an alternative embodiment, the compressor logic 404allocates bits to achieve higher quality in a particular tile or groupof tiles. For example, it may provide a better overall perceptualappearance to have higher quality in the center of the screen than atthe edges.

In one embodiment, to improve resolution of certain regions of the videostream, the video compression logic 404 uses smaller tiles to encodeareas of the video stream with relatively more scene complexity and/ormotion than areas of the video stream with relatively less scenecomplexity and/or motion. For example, as illustrated in FIG. 8 ,smaller tiles are employed around a moving character 805 in one area ofone R frame 811 (potentially followed by a series of R frames with thesame tile sizes (not shown)). Then, when the character 805 moves to anew area of the image, smaller tiles are used around this new areawithin another R frame 812, as illustrated. As mentioned above, variousdifferent sizes and shapes may be employed as “tiles” while stillcomplying with these underlying principles.

While the cyclic VP tiles described above substantially reduce the peaksin the data rate of a video stream, they do not eliminate the peaksentirely, particularly in the case of rapidly-changing or highly complexvideo imagery, such as occurs with motion pictures, video games, andsome application software. For example, during a sudden scenetransition, a complex frame may be followed by another complex framethat is completely different. Even though several I tiles may havepreceded the scene transition by only a few frame times, they don't helpin this situation because the new frame's material has no relation tothe previous I tiles. In such a situation (and in other situations whereeven though not everything changes, much of the image changes), thevideo compressor 404 will determine that many, if not all, of the Ptiles are more efficiently coded as I tiles, and what results is a verylarge peak in the data rate for that frame.

As discussed previously, it is simply the case that with mostconsumer-grade Internet connections (and many office connections), itsimply is not feasible to “jam” data that exceeds the available maximumdata rate shown as 622 in FIG. 6 c , along with the rated maximum datarate 621. Note that the rated maximum data rate 621 (e.g., “6 Mbps DSL”)is essentially a marketing number for users considering the purchase ofan Internet connection, but generally it does not guarantee a level ofperformance. For the purposes of this application, it is irrelevant,since our only concern is the available maximum data rate 622 at thetime the video is streamed through the connection. Consequently, inFIGS. 9 a and 9 c , as we describe a solution to the peaking problem,the rated maximum data rate is omitted from the graph, and only theavailable maximum data rate 922 is shown. The video stream data ratemust not exceed the available maximum data rate 922.

To address this, the first thing that the video compressor 404 does isdetermine a peak data rate 941, which is a data rate the channel is ableto handle steadily. This rate can be determined by a number oftechniques. One such technique is by gradually sending an increasinglyhigher data rate test stream from the hosting service 210 to the client415 in FIGS. 4 a and 4 b , and having the client provide feedback to thehosting service as to the level of packet loss and latency. As thepacket loss and/or latency begins to show a sharp increase, that is anindication that the available maximum data rate 922 is being reached.After that, the hosting service 210 can gradually reduce the data rateof the test stream until the client 415 reports that for a reasonableperiod of time the test stream has been received with an acceptablelevel of packet loss and the latency is near minimal. This establishes apeak maximum data rate 941, which will then be used as a peak data ratefor streaming video. Over time, the peak data rate 941 will fluctuate(e.g., if another user in a household starts to heavily use the Internetconnection), and the client 415 will need to constantly monitor it tosee whether packet loss or latency increases, indicating the availablemax data rate 922 is dropping below the previously established peak datarate 941, and if so the peak data rate 941. Similarly, if over time theclient 415 finds that the packet loss and latency remain at optimallevels, it can request that the video compressor slowly increases thedata rate to see whether the available maximum data rate has increased(e.g., if another user in a household has stopped heavy use of theInternet connection), and again waiting until packet loss and/or higherlatency indicates that the available maximum data rate 922 has beenexceeded, and again a lower level can be found for the peak data rate941, but one that is perhaps higher than the level before testing anincreased data rate. So, by using this technique (and other techniqueslike it) a peak data rate 941 can be found, and adjusted periodically asneeded. The peak data rate 941 will establishes the maximum data ratethat can be used by the video compressor 404 to stream video to theuser. The logic for determining the peak data rate may be implemented atthe user premises 211 and/or on the hosting service 210. At the userpremises 211, the client device 415 performs the calculations todetermine the peak data rate and transmits this information back to thehosting service 210; at the hosting service 210, a server 402 at thehosting service performs the calculations to determine the peak datarate based on statistics received from the client 415 (e.g., packetloss, latency, max data rate, etc).

FIG. 9 a shows an example video stream data rate 934 that hassubstantial scene complexity and/or motion that has been generated usingthe cyclic I/P tile compression techniques described previously andillustrated in FIGS. 7 a, 7 b and 8. The video compressor 404 has beenconfigured to output compressed video at an average data rate that isbelow the peak data rate 941, and note that, most of the time, the videostream data rate remains below the peak data rate 941. A comparison ofdata rate 934 with video stream data rate 634 shown in FIG. 6 c createdusing I/P/B or I/P frames shows that the cyclic I/P tile compressionproduces a much smoother data rate. Still, at frame 2× peak 952 (whichapproaches 2× the peak data rate 942) and frame 4× peak 954 (whichapproaches 4× the peak data rate 944), the data rate exceeds the peakdata rate 941, which is unacceptable. In practice, even with high actionvideo from rapidly changing video games, peaks in excess of peak datarate 941 occur in less than 2% of frames, peaks in excess of 2× peakdata rate 942 occur rarely, and peaks in excess of 3× peak data rate 943occur hardly ever. But, when they do occur (e.g., during a scenetransition), the data rate required by them is necessary to produce agood quality video image.

One way to solve this problem is simply to configure the videocompressor 404 such that its maximum data rate output is the peak datarate 941. Unfortunately, the resulting video output quality during thepeak frames is poor since the compression algorithm is “starved” forbits. What results is the appearance of compression artifacts when thereare sudden transitions or fast motion, and in time, the user comes torealize that the artifacts always crop up when there is sudden changesor rapid motion, and they can become quite annoying.

Although the human visual system is quite sensitive to visual artifactsthat appear during sudden changes or rapid motion, it is not verysensitive to detecting a reduction in frame rate in such situations. Infact, when such sudden changes occur, it appears that the human visualsystem is preoccupied with tracking the changes, and it doesn't noticeif the frame rate briefly drops from 60 fps to 30 fps, and then returnsimmediately to 60 fps. And, in the case of a very dramatic transition,like a sudden scene change, the human visual system does not notice ifthe frame rate drops to 20 fps or even 15 fps, and then immediatelyreturns to 60 fps. So long as the frame rate reduction only occursinfrequently, to a human observer, it appears that the video has beencontinuously running at 60 fps.

This property of the human visual system is exploited by the techniquesillustrated in FIG. 9 b . A server 402 (from FIGS. 4 a and 4 b )produces an uncompressed video output stream at a steady frame rate (at60 fps in one embodiment). A timeline shows each frame 961-970 outputeach 1/60^(th) second. Each uncompressed video frame, starting withframe 961, is output to the low-latency video compressor 404, whichcompresses the frame in less than a frame time, producing for the firstframe compressed frame 1 981. The data produced for the compressed frame1 981 may be larger or smaller, depending upon many factors, aspreviously described. If the data is small enough that it can betransmitted to the client 415 in a frame time ( 1/60^(th) second) orless at the peak data rate 941, then it is transmitted during transmittime (xmit time) 991 (the length of the arrow indicates the duration ofthe transmit time). In the next frame time, server 402 producesuncompressed frame 2 962, it is compressed to compressed frame 2 982,and it is transmitted to client 415 during transmit time 992, which isless than a frame time at peak data rate 941.

Then, in the next frame time, server 402 produces uncompressed frame 3963. When it is compressed by video compressor 404, the resultingcompressed frame 3 983 is more data than can be transmitted at the peakdata rate 941 in one frame time. So, it is transmitted during transmittime (2× peak) 993, which takes up all of the frame time and part of thenext frame time. Now, during the next frame time, server 402 producesanother uncompressed frame 4 964 and outputs it to video compressor 404but the data is ignored and illustrated with 974. This is because videocompressor 404 is configured to ignore further uncompressed video framesthat arrive while it is still transmitting a prior compressed frame. Ofcourse client 415's video decompressor will fail to receive frame 4, butit simply continues to display on display device 422 frame 3 for 2 frametimes (i.e., briefly reduces the frame rate from 60 fps to 30 fps).

For the next frame 5, server 402 outputs uncompressed frame 5 965, iscompressed to compressed frame 5 985 and transmitted within 1 frameduring transmit time 995. Client 415's video decompressor decompressesframe 5 and displays it on display device 422. Next, server 402 outputsuncompressed frame 6 966, video compressor 404 compresses it tocompressed frame 6 986, but this time the resulting data is very large.The compressed frame is transmitted during transmit time (4× peak) 996at the peak data rate 941, but it takes almost 4 frame times to transmitthe frame. During the next 3 frame times, video compressor 404 ignores 3frames from server 402, and client 415's decompressor holds frame 6steadily on the display device 422 for 4 frames times (i.e., brieflyreduces the frame rate from 60 fps to 15 fps). Then finally, server 402outputs frame 10 970, video compressor 404 compresses it into compressedframe 10 987, and it is transmitted during transmit time 997, and client415's decompressor decompresses frame 10 and displays it on displaydevice 422 and once again the video resumes at 60 fps.

Note that although video compressor 404 drops video frames from thevideo stream generated by server 402, it does not drop audio data,regardless of what form the audio comes in, and it continues to compressthe audio data when video frames are dropped and transmit them to client415, which continues to decompress the audio data and provide the audioto whatever device is used by the user to playback the audio. Thus audiocontinues unabated during periods when frames are dropped. Compressedaudio consumes a relatively small percentage of bandwidth, compared tocompressed video, and as result does not have a major impact on theoverall data rate. Although it is not illustrated in any of the datarate diagrams, there is always data rate capacity reserved for thecompressed audio stream within the peak data rate 941.

The example just described in FIG. 9 b was chosen to illustrate how theframe rate drops during data rate peaks, but what it does not illustrateis that when the cyclic I/P tile techniques described previously areused, such data rate peaks, and the consequential dropped frames arerare, even during high scene complexity/high action sequences such asthose that occur in video games, motion pictures and some applicationsoftware. Consequently, the reduced frame rates are infrequent andbrief, and the human visual system does not detect them.

If the frame rate reduction mechanism just described is applied to thevideo stream data rate illustrated in FIG. 9 a , the resulting videostream data rate is illustrated in FIG. 9 c . In this example, 2× peak952 has been reduced to flattened 2× peak 953, and 4× peak 955 has beenreduced to flattened 4× peak 955, and the entire video stream data rate934 remains at or below the peak data rate 941.

Thus, using the techniques described above, a high action video streamcan be transmitted with low latency through the general Internet andthrough a consumer-grade Internet connection. Further, in an officeenvironment on a LAN (e.g., 100 Mbs Ethernet or 802.11g wireless) or ona private network (e.g., 100 Mbps connection between a data center anoffices) a high action video stream can be transmitted without peaks sothat multiple users (e.g., transmitting 1920×1080 at 60 fps at 4.5 Mbps)can use the LAN or shared private data connection without havingoverlapping peaks overwhelming the network or the network switchbackplanes.

Data Rate Adjustment

In one embodiment, the hosting service 210 initially assesses theavailable maximum data rate 622 and latency of the channel to determinean appropriate data rate for the video stream and then dynamicallyadjusts the data rate in response. To adjust the data rate, the hostingservice 210 may, for example, modify the image resolution and/or thenumber of frames/second of the video stream to be sent to the client415. Also, the hosting service can adjust the quality level of thecompressed video. When changing the resolution of the video stream,e.g., from a 1280×720 resolution to a 640×360 the video decompressionlogic 412 on the client 415 can scale up the image to maintain the sameimage size on the display screen.

In one embodiment, in a situation where the channel completely dropsout, the hosting service 210 pauses the game. In the case of amultiplayer game, the hosting service reports to the other users thatthe user has dropped out of the game and/or pauses the game for theother users.

Dropped or Delayed Packets

In one embodiment, if data is lost due to packet loss between the videocompressor 404 and client 415 in FIG. 4 a or 4 b, or due to a packetbeing received out of order that arrives too late to decompress and meetthe latency requirements of the decompressed frame, the videodecompression logic 412 is able to mitigate the visual artifacts. In astreaming I/P frame implementation, if there is a lost/delayed packet,the entire screen is impacted, potentially causing the screen tocompletely freeze for a period of time or show other screen-wide visualartifacts. For example, if a lost/delayed packet causes the loss of an Iframe, then the decompressor will lack a reference for all of the Pframes that follow until a new I frame is received. If a P frame islost, then it will impact the P frames for the entire screen thatfollow. Depending on how long it will be before an I frame appears, thiswill have a longer or shorter visual impact. Using interleaved I/P tilesas shown in FIGS. 7 a and 7 b , a lost/delayed packet is much lesslikely to impact the entire screen since it will only affect the tilescontained in the affected packet. If each tile's data is sent within anindividual packet, then if a packet is lost, it will only affect onetile. Of course, the duration of the visual artifact will depend onwhether an I tile packet is lost and, if a P tile is lost, how manyframes it will take until an I tile appears. But, given that differenttiles on the screen are being updated with I frames very frequently(potentially every frame), even if one tile on the screen is affected,other tiles may not be. Further, if some event cause a loss of severalpackets at once (e.g., spike in power next to a DSL line that brieflydisrupts the data flow), then some of the tiles will be affected morethan others, but because some tiles will quickly be renewed with a new Itile, they will be only briefly affected. Also, with a streaming I/Pframe implementation, not only are the I frames the most critical frame,but the I frames are extremely large, so if there is an event thatcauses a dropped/delayed packet, there is a higher probability that an Iframe will be affected (i.e., if any part of an I frame is lost, it isunlikely that the I frame can be decompressed at all) than a muchsmaller I tile. For all of these reasons, using I/P tiles results in farfewer visual artifacts when packets are dropped/delayed than with I/Pframes.

One embodiment attempts to reduce the effect of lost packets byintelligently packaging the compressed tiles within the TCP(transmission control protocol) packets or UDP (user datagram protocol)packets. For example, in one embodiment, tiles are aligned with packetboundaries whenever possible. FIG. 10 a illustrates how tiles might bepacked within a series of packets 1001-1005 without implementing thisfeature. Specifically, in FIG. 10 a , tiles cross packet boundaries andare packed inefficiently so that the loss of a single packet results inthe loss of multiple frames. For example, if packets 1003 or 1004 arelost, three tiles are lost, resulting in visual artifacts.

By contrast, FIG. 10 b illustrates tile packing logic 1010 forintelligently packing tiles within packets to reduce the effect ofpacket loss. First, the tile packing logic 1010 aligns tiles with packetboundaries. Thus, tiles T1, T3, T4, T7, and T2 are aligned with theboundaries of packets 1001-1005, respectively. The tile packing logicalso attempts to fit tiles within packets in the most efficient mannerpossible, without crossing packet boundaries. Based on the size of eachof the tiles, tiles T1 and T6 are combined in one packet 1001; T3 and T5are combined in one packet 1002; tiles T4 and T8 are combined in onepacket 1003; tile T8 is added to packet 1004; and tile T2 is added topacket 1005. Thus, under this scheme, a single packet loss will resultin the loss of no more than 2 tiles (rather than 3 tiles as illustratedin FIG. 10 a ).

One additional benefit to the embodiment shown in FIG. 10 b is that thetiles are transmitted in a different order in which they are displayedwithin the image. This way, if adjacent packets are lost from the sameevent interfering with the transmission it will affect areas which arenot near each other on the screen, creating a less noticeableartifacting on the display.

One embodiment employs forward error correction (FEC) techniques toprotect certain portions of the video stream from channel errors. As isknown in the art, FEC techniques such as Reed-Solomon and Viterbigenerate and append error correction data information to datatransmitted over a communications channel. If an error occurs in theunderlying data (e.g., an I frame), then the FEC may be used to correctthe error.

FEC codes increase the data rate of the transmission; so ideally, theyare only used where they are most needed. If data is being sent thatwould not result in a very noticeable visual artifact, it may bepreferable to not use FEC codes to protect the data. For example, a Ptile that immediately precedes an I tile that is lost will only create avisual artifact (i.e., on tile on the screen will not be updated) for1/60^(th) of second on the screen. Such a visual artifact is barelydetectable by the human eye. As P tiles are further back from an I tile,losing a P tile becomes increasingly more noticeable. For example, if atile cycle pattern is an I tile followed by 15 P tiles before an I tileis available again, then if the P tile immediately following an I tileis lost, it will result in that tile showing an incorrect image for 15frame times (at 60 fps, that would be 250 ms). The human eye willreadily detect a disruption in a stream for 250 ms. So, the further backa P tile is from a new I tile (i.e., the closer a P tiles follows an Itile), the more noticeable the artifact. As previously discussed,though, in general, the closer a P tile follows an I tile, the smallerthe data for that P tile. Thus, P tiles following I tiles not only aremore critical to protect from being lost, but they are smaller in size.And, in general, the smaller the data is that needs to be protected, thesmaller the FEC code needs to be to protect it.

So, as illustrated in FIG. 11 a , in one embodiment, because of theimportance of I tiles in the video stream, only I tiles are providedwith FEC codes. Thus, FEC 1101 contains error correction code for I tile1100 and FEC 1104 contains error correction code for I tile 1103. Inthis embodiment, no FEC is generated for the P tiles.

In one embodiment illustrated in FIG. 11 b FEC codes are also generatedfor P tiles which are most likely to cause visual artifacts if lost. Inthis embodiment, FECs 1105 provide error correction codes for the first3 P tiles, but not for the P tiles that follow. In another embodiment,FEC codes are generated for P tiles which are smallest in data size(which will tend to self-select P tiles occurring the soonest after an Itile, which are the most critical to protect).

In another embodiment, rather than sending an FEC code with a tile, thetile is transmitted twice, each time in a different packet. If onepacket is lost/delayed, the other packet is used.

In one embodiment, shown in FIG. 11 c , FEC codes 1111 and 1113 aregenerated for audio packets, 1110 and 1112, respectively, transmittedfrom the hosting service concurrently with the video. It is particularlyimportant to maintain the integrity of the audio in a video streambecause distorted audio (e.g., clicking or hissing) will result in aparticularly undesirable user experience. The FEC codes help to ensurethat the audio content is rendered at the client computer 415 withoutdistortion.

In another embodiment, rather than sending an FEC code with audio data,the audio data is transmitted twice, each time in a different packet. Ifone packet is lost/delayed, the other packet is used.

In addition, in one embodiment illustrated in FIG. 11 d , FEC codes 1121and 1123 are used for user input commands 1120 and 1122, respectively(e.g., button presses) transmitted upstream from the client 415 to thehosting service 210. This is important because missing a button press ora mouse movement in a video game or an application could result in anundesirable user experience.

In another embodiment, rather than sending an FEC code with user inputcommand data, the user input command data is transmitted twice, eachtime in a different packet. If one packet is lost/delayed, the otherpacket is used.

In one embodiment, the hosting service 210 assesses the quality of thecommunication channel with the client 415 to determine whether to useFEC and, if so, what portions of the video, audio and user commands towhich FEC should be applied. Assessing the “quality” of the channel mayinclude functions such as evaluating packet loss, latency, etc, asdescribed above. If the channel is particularly unreliable, then thehosting service 210 may apply FEC to all of I tiles, P tiles, audio anduser commands. By contrast, if the channel is reliable, then the hostingservice 210 may apply FEC only to audio and user commands, or may notapply FEC to audio or video, or may not use FEC at all. Various otherpermutations of the application of FEC may be employed while stillcomplying with these underlying principles. In one embodiment, thehosting service 210 continually monitors the conditions of the channeland changes the FEC policy accordingly.

In another embodiment, referring to FIGS. 4 a and 4 b , when a packet islost/delayed resulting in the loss of tile data or if, perhaps becauseof a particularly bad packet loss, the FEC is unable to correct losttile data, the client 415 assesses how many frames are left before a newI tile will be received and compares it to the round-trip latency fromthe client 415 to hosting service 210. If the round-trip latency is lessthan the number of frames before a new I tile is due to arrive, then theclient 415 sends a message to the hosting service 210 requesting a new Itile. This message is routed to the video compressor 404, and ratherthan generating a P tile for the tile whose data had been lost, itgenerates an I tile. Given that the system shown in FIGS. 4 a and 4 b isdesigned to provide a round-trip latency that is typically less than 80ms, this results in a tile being corrected within 80 ms (at 60 fps,frames are 16.67 ms of duration, thus in full frame times, 80 ms latencywould result in a corrected a tile within 83.33 ms, which is 5 frametimes—a noticeable disruption, but far less noticeable than, forexample, a 250 ms disruption for 15 frames). When the compressor 404generates such an I tile out of its usual cyclic order, if the I tilewould cause the bandwidth of that frame to exceed the availablebandwidth, then the compressor 404 will delay the cycles of the othertiles so that the other tiles receive P tiles during that frame time(even if one tile would normally be due an I tile during that frame),and then starting with the next frame the usual cycling will continue,and the tile that normally would have received an I tile in thepreceding frame will receive an I tile. Although this action brieflydelays the phase of the R frame cycling, it normally will not benoticeable visually.

Video and Audio Compressor/Decompressor Implementation

FIG. 12 illustrates one particular embodiment in which a multi-coreand/or multi-processor 1200 is used to compress 8 tiles in parallel. Inone embodiment, a dual processor, quad core Xeon CPU computer systemrunning at 2.66 GHz or higher is used, with each core implementing theopen source x264 H.264 compressor as an independent process. However,various other hardware/software configurations may be used while stillcomplying with these underlying principles. For example, each of the CPUcores can be replaced with an H.264 compressor implemented in an FPGA.In the example shown in FIG. 12 , cores 1201-1208 are used toconcurrently process the I tiles and P tiles as eight independentthreads. As is well known in the art, current multi-core andmulti-processor computer systems are inherently capable ofmulti-threading when integrated with multi-threading operating systemssuch as Microsoft Windows XP Professional Edition (either 64-bit or the32-bit edition) and Linux.

In the embodiment illustrated in FIG. 12 , since each of the 8 cores isresponsible for just one tile, it operates largely independently fromthe other cores, each running a separate instantiation of x264. A PCIExpress x1-based DVI capture card, such as the Sendero Video Imaging IPDevelopment Board from Microtronix of Oosterhout, The Netherlands isused to capture uncompressed video at 640×480, 800×600, or 1280×720resolution, and the FPGA on the card uses Direct Memory Access (DMA) totransfer the captured video through the DVI bus into system RAM. Thetiles are arranged in a 4×2 arrangement 1205 (although they areillustrated as square tiles, in this embodiment they are of 160×240resolution). Each instantiation of x264's is configured to compress oneof the 8 160×240 tiles, and they are synchronized such that, after aninitial I tile compression, each core enters into a cycle, each oneframe out of phase with the other, to compress one I tile followed byseven P tiles, and illustrated in FIG. 12 .

Each frame time, the resulting compressed tiles are combined into apacket stream, using the techniques previously described, and then thecompressed tiles are transmitted to a destination client 415.

Although not illustrated in FIG. 12 , if the data rate of the combined 8tiles exceeds a specified peak data rate 941, then all 8×264 processesare suspended for as many frame times as are necessary until the datafor the combined 8 tiles has been transmitted.

In one embodiment, client 415 is implemented as software on a PC running8 instantiations of FFmpeg. A receiving process receives the 8 tiles,and each tile is routed to an FFmpeg instantiation, which decompressesthe tile and renders it to an appropriate tile location on the displaydevice 422.

The client 415 receives keyboard, mouse, or game controller input fromthe PC's input device drivers and transmits it to the server 402. Theserver 402 then applies the received input device data and applies it tothe game or application running on the server 402, which is a PC runningWindows using an Intel 2.16 GHz Core Duo CPU. The server 402 thenproduces a new frame and outputs it through its DVI output, either froma motherboard-based graphics system, or through a NVIDIA 8800GTX PCIcard's DVI output.

Simultaneously, the server 402 outputs the audio produced by game orapplications through its digital audio output (e.g., S/PDIF), which iscoupled to the digital audio input on the dual quad-core Xeon-based PCthat is implementing the video compression. A Vorbis open source audiocompressor is used to compress the audio simultaneously with the videousing whatever core is available for the process thread. In oneembodiment, the core that completes compressing its tile first executesthe audio compression. The compressed audio is then transmitted alongwith the compressed video, and is decompressed on the client 415 using aVorbis audio decompressor.

Hosting Service Server Center Distribution

Light through glass, such as optical fiber, travels at some fraction ofthe speed of light in a vacuum, and so an exact propagation speed forlight in optical fiber could be determined. But, in practice, allowingtime for routing delays, transmission inefficiencies, and otheroverhead, we have observed that optimal latencies on the Internetreflect transmission speeds closer to 50% the speed of light. Thus, anoptimal 1000 mile round trip latency is approximately 22 ms, and anoptimal 3000 mile round trip latency is about 64 ms. Thus, a singleserver on one US coast will be too far away to serve clients on theother coast (which can be as far as 3000 miles away) with the desiredlatency. However, as illustrated in FIG. 13 a , if the hosting service210 server center 1300 is located in the center of the US (e.g., Kansas,Nebraska, etc.), such that the distance to any point in the continentalUS is approximately 1500 miles or less, the round trip Internet latencycould be as low as 32 ms. Referring to FIG. 4 b , note that although theworst-case latencies allowed for the user ISP 453 is 25 ms, typically,we have observed latencies closer to 10-15 ms with DSL and cable modemsystems. Also, FIG. 4 b assumes a maximum distance from the userpremises 211 to the hosting center 210 of 1000 miles. Thus, with atypical user ISP round trip latency of 15 ms used and a maximum Internetdistance of 1500 miles for a round trip latency of 32 ms, the totalround trip latency from the point a user actuates input device 421 andsees a response on display device 422 is 1+1+15+32+1+16+6+8=80 ms. So,the 80 ms response time can be typically achieved over an Internetdistance of 1500 miles. This would allow any user premises with a shortenough user ISP latency 453 in the continental US to access a singleserver center that is centrally located.

In another embodiment, illustrated in FIG. 13 b , the hosting service210 server centers, HS1-HS6, are strategically positioned around theUnited States (or other geographical region), with certain largerhosting service server centers positioned close to high populationcenters (e.g., HS2 and HS5). In one embodiment, the server centersHS1-HS6 exchange information via a network 1301 which may be theInternet or a private network or a combination of both. With multipleserver centers, services can be provided at lower latency to users thathave high user ISP latency 453.

Although distance on the Internet is certainly a factor that contributesto round trip latency through the Internet, sometimes other factors comeinto play that are largely unrelated to latency. Sometimes a packetstream is routed through the Internet to a far away location and backagain, resulting in latency from the long loop. Sometimes there isrouting equipment on the path that is not operating properly, resultingin a delay of the transmission. Sometimes there is a traffic overloadinga path which introduces delay. And, sometimes, there is a failure thatprevents the user's ISP from routing to a given destination at all.Thus, while the general Internet usually provides connections from onepoint to another with a fairly reliable and optimal route and latencythat is largely determined by distance (especially with long distanceconnections that result in routing outside of the user's local area)such reliability and latency is by no means guaranteed and often cannotbe achieved from a user's premises to a given destination on the generalInternet.

In one embodiment, when a user client 415 initially connects to thehosting service 210 to play a video game or use an application, theclient communicates with each of the hosting service server centersHS1-HS6 available upon startup (e.g., using the techniques describedabove). If the latency is low enough for a particular connection, thenthat connection is used. In one embodiment, the client communicates withall, or a subset, of the hosting service server centers the one with thelowest latency connection is selected. The client may select the servicecenter with the lowest latency connection or the service centers mayidentify the one with the lowest latency connection and provide thisinformation (e.g., in the form of an Internet address) to the client.

If a particular hosting service server center is overloaded and/or theuser's game or application can tolerate the latency to another, lessloaded hosting service server center, then the client 415 may beredirected to the other hosting service server center. In such asituation, the game or application the user is running would be pausedon the server 402 at the user's overloaded server center, and the gameor application state data would be transferred to a server 402 atanother hosting service server center. The game or application wouldthen be resumed. In one embodiment, the hosting service 210 would waituntil the game or application has either reached a natural pausing point(e.g., between levels in a game, or after the user initiates a “save”operation in application) to do the transfer. In yet another embodiment,the hosting service 210 would wait until user activity ceases for aspecified period of time (e.g., 1 minute) and then would initiate thetransfer at that time.

As described above, in one embodiment, the hosting service 210subscribes to an Internet bypass service 440 of FIG. 14 to attempt toprovide guaranteed latency to its clients. Internet bypass services, asused herein, are services that provide private network routes from onepoint to another on the Internet with guaranteed characteristics (e.g.,latency, data rate, etc.). For example, if the hosting service 210 wasreceiving large amount of traffic from users using AT&T's DSL serviceoffering in San Francisco, rather than routing to AT&T's SanFrancisco-based central offices, the hosting service 210 could lease ahigh-capacity private data connection from a service provider (perhapsAT&T itself or another provider) between the San Francisco-based centraloffices and one or more of the server centers for hosting service 210.Then, if routes from all hosting service server centers HS1-HS6 throughthe general Internet to a user in San Francisco using AT&T DSL result intoo high latency, then private data connection could be used instead.Although private data connections are generally more expensive than theroutes through the general Internet, so long as they remain a smallpercentage of the hosting service 210 connections to users, the overallcost impact will be low, and users will experience a more consistentservice experience.

Server centers often have two layers of backup power in the event ofpower failure. The first layer typically is backup power from batteries(or from an alternative immediately available energy source, such aflywheel that is kept running and is attached to a generator), whichprovides power immediately when the power mains fail and keeps theserver center running. If the power failure is brief, and the powermains return quickly (e.g., within a minute), then the batteries are allthat is needed to keep the server center running. But if the powerfailure is for a longer period of time, then typically generators (e.g.,diesel-powered) are started up that take over for the batteries and canrun for as long as they have fuel. Such generators are extremelyexpensive since they must be capable of producing as much power as theserver center normally gets from the power mains.

In one embodiment, each of the hosting services HS1-HS5 share user datawith one another so that if one server center has a power failure, itcan pause the games and applications that are in process, and thentransfer the game or application state data from each server 402 toservers 402 at other server centers, and then will notify the client 415of each user to direct it communications to the new server 402. Giventhat such situations occur infrequently, it may be acceptable totransfer a user to a hosting service server center which is not able toprovide optimal latency (i.e., the user will simply have to toleratehigher latency for the duration of the power failure), which will allowfor a much wider range of options for transferring users. For example,given the time zone differences across the US, users on the East Coastmay be going to sleep at 11:30 PM while users on the West Coast at 8:30PM are starting to peak in video game usage. If there is a power failurein a hosting service server center on the West Coast at that time, theremay not be enough West Coast servers 402 at other hosting service servercenters to handle all of the users. In such a situation, some of theusers can be transferred to hosting service server centers on the EastCoast which have available servers 402, and the only consequence to theusers would be higher latency. Once the users have been transferred fromthe server center that has lost power, the server center can thencommence an orderly shutdown of its servers and equipment, such that allof the equipment has been shut down before the batteries (or otherimmediate power backup) is exhausted. In this way, the cost of agenerator for the server center can be avoided.

In one embodiment, during times of heavy loading of the hosting service210 (either due to peak user loading, or because one or more servercenters have failed) users are transferred to other server centers onthe basis of the latency requirements of the game or application theyare using. So, users using games or applications that require lowlatency would be given preference to available low latency serverconnections when there is a limited supply.

Hosting Service Features

FIG. 15 illustrates an embodiment of components of a server center forhosting service 210 utilized in the following feature descriptions. Aswith the hosting service 210 illustrated in FIG. 2 a , the components ofthis server center are controlled and coordinated by a hosting service210 control system 401 unless otherwise qualified.

Inbound internet traffic 1501 from user clients 415 is directed toinbound routing 1502. Typically, inbound internet traffic 1501 willenter the server center via a high-speed fiber optic connection to theInternet, but any network connection means of adequate bandwidth,reliability and low latency will suffice. Inbound routing 1502 is asystem of network (the network can be implemented as an Ethernetnetwork, a fiber channel network, or through any other transport means)switches and routing servers supporting the switches which takes thearriving packets and routes each packet to the appropriateapplication/game (“app/game”) server 1521-1525. In one embodiment, apacket which is delivered to a particular app/game server represents asubset of the data received from the client and/or may betranslated/changed by other components (e.g., networking components suchas gateways and routers) within the data center. In some cases, packetswill be routed to more than one server 1521-1525 at a time, for example,if a game or application is running on multiple servers at once inparallel. RAID array 1511-1512 are connected to the inbound routingnetwork 1502, such that the app/game servers 1521-1525 can read andwrite to the RAID arrays 1511-1512. Further, a RAID array 1515 (whichmay be implemented as multiple RAID arrays) is also connected to theinbound routing 1502 and data from RAID array 1515 can be read fromapp/game servers 1521-1525. The inbound routing 1502 may be implementedin a wide range of prior art network architectures, including a treestructure of switches, with the inbound internet traffic 1501 at itsroot; in a mesh structure interconnecting all of the various devices; oras an interconnected series of subnets, with concentrated trafficamongst intercommunicating device segregated from concentrated trafficamongst other devices. One type of network configuration is a SAN which,although typically used for storage devices, it can also be used forgeneral high-speed data transfer among devices. Also, the app/gameservers 1521-1525 may each have multiple network connections to theinbound routing 1502. For example, a server 1521-1525 may have a networkconnection to a subnet attached to RAID Arrays 1511-1512 and anothernetwork connection to a subnet attached to other devices.

The app/game servers 1521-1525 may all be configured the same, somedifferently, or all differently, as previously described in relation toservers 402 in the embodiment illustrated in FIG. 4 a . In oneembodiment, each user, when using the hosting service is typically atleast one app/game server 1521-1525. For the sake of simplicity ofexplanation, we shall assume a given user is using app/game server 1521,but multiple servers could be used by one user, and multiple users couldshare a single app/game server 1521-1525. The user's control input, sentfrom client 415 as previously described is received as inbound Internettraffic 1501, and is routed through inbound routing 1502 to app/gameserver 1521. App/game server 1521 uses the user's control input ascontrol input to the game or application running on the server, andcomputes the next frame of video and the audio associated with it.App/game server 1521 then outputs the uncompressed video/audio 1529 toshared video compression 1530. App/game server may output theuncompressed video via any means, including one or more Gigabit Ethernetconnections, but in one embodiment the video is output via a DVIconnection and the audio and other compression and communication channelstate information is output via a Universal Serial Bus (USB) connection.

The shared video compression 1530 compresses the uncompressed video andaudio from the app/game servers 1521-1525. The compression maybeimplemented entirely in hardware, or in hardware running software. Theremay a dedicated compressor for each app/game server 1521-1525, or if thecompressors are fast enough, a given compressor can be used to compressthe video/audio from more than one app/game server 1521-1525. Forexample, at 60 fps a video frame time is 16.67 ms. If a compressor isable to compress a frame in 1 ms, then that compressor could be used tocompress the video/audio from as many as 16 app/game servers 1521-1525by taking input from one server after another, with the compressorsaving the state of each video/audio compression process and switchingcontext as it cycles amongst the video/audio streams from the servers.This results in substantial cost savings in compression hardware. Sincedifferent servers will be completing frames at different times, in oneembodiment, the compressor resources are in a shared pool 1530 withshared storage means (e.g., RAM, Flash) for storing the state of eachcompression process, and when a server 1521-1525 frame is complete andready to be compressed, a control means determines which compressionresource is available at that time, provides the compression resourcewith the state of the server's compression process and the frame ofuncompressed video/audio to compress.

Note that part of the state for each server's compression processincludes information about the compression itself, such as the previousframe's decompressed frame buffer data which may be used as a referencefor P tiles, the resolution of the video output; the quality of thecompression; the tiling structure; the allocation of bits per tiles; thecompression quality, the audio format (e.g., stereo, surround sound,Dolby® AC-3). But the compression process state also includescommunication channel state information regarding the peak data rate 941and whether a previous frame (as illustrated in FIG. 9 b ) is currentlybeing output (and as result the current frame should be ignored), andpotentially whether there are channel characteristics which should beconsidered in the compression, such as excessive packet loss, whichaffect decisions for the compression (e.g., in terms of the frequency ofI tiles, etc). As the peak data rate 941 or other channelcharacteristics change over time, as determined by an app/game server1521-1525 supporting each user monitoring data sent from the client 415,the app/game server 1521-1525 sends the relevant information to theshared hardware compression 1530.

The shared hardware compression 1530 also packetizes the compressedvideo/audio using means such as those previously described, and ifappropriate, applying FEC codes, duplicating certain data, or takingother steps to as to adequately ensure the ability of the video/audiodata stream to be received by the client 415 and decompressed with ashigh a quality and reliability as feasible.

Some applications, such as those described below, require thevideo/audio output of a given app/game server 1521-1525 to be availableat multiple resolutions (or in other multiple formats) simultaneously.If the app/game server 1521-1525 so notifies the shared hardwarecompression 1530 resource, then the uncompressed video audio 1529 ofthat app/game server 1521-1525 will be simultaneously compressed indifferent formats, different resolutions, and/or in differentpacket/error correction structures. In some cases, some compressionresources can be shared amongst multiple compression processescompressing the same video/audio (e.g., in many compression algorithms,there is a step whereby the image is scaled to multiple sizes beforeapplying compression. If different size images are required to beoutput, then this step can be used to serve several compressionprocesses at once). In other cases, separate compression resources willbe required for each format. In any case, the compressed video/audio1539 of all of the various resolutions and formats required for a givenapp/game server 1521-1525 (be it one or many) will be output at once tooutbound routing 1540. In one embodiment the output of the compressedvideo/audio 1539 is in UDP format, so it is a unidirectional stream ofpackets.

The outbound routing network 1540 comprises a series of routing serversand switches which direct each compressed video/audio stream to theintended user(s) or other destinations through outbound Internet traffic1599 interface (which typically would connect to a fiber interface tothe Internet) and/or back to the delay buffer 1515, and/or back to theinbound routing 1502, and/or out through a private network (not shown)for video distribution. Note that (as described below) the outboundrouting 1540 may output a given video/audio stream to multipledestinations at once. In one embodiment this is implemented usingInternet Protocol (IP) multicast in which a given UDP stream intended tobe streamed to multiple destinations at once is broadcasted, and thebroadcast is repeated by the routing servers and switches in theoutbound routing 1540. The multiple destinations of the broadcast may beto multiple users' clients 415 via the Internet, to multiple app/gameservers 1521-1525 through via inbound routing 1502, and/or to one ormore delay buffers 1515. Thus, the output of a given server 1521-1522 iscompressed into one or multiple formats, and each compressed stream isdirected to one or multiple destinations.

Further, in another embodiment, if multiple app/game servers 1521-1525are used simultaneously by one user (e.g., in a parallel processingconfiguration to create the 3D output of a complex scene) and eachserver is producing part of the resulting image, the video output ofmultiple servers 1521-1525 can be combined by the shared hardwarecompression 1530 into a combined frame, and from that point forward itis handled as described above as if it came from a single app/gameserver 1521-1525.

Note that in one embodiment, a copy (in at least the resolution orhigher of video viewed by the user) of all video generated by app/gameservers 1521-1525 is recorded in delay buffer 1515 for at least somenumber of minutes (15 minutes in one embodiment). This allows each userto “rewind” the video from each session in order to review previous workor exploits (in the case of a game). Thus, in one embodiment, eachcompressed video/audio output 1539 stream being routed to a user client415 is also being multicasted to a delay buffer 1515. When thevideo/audio is stored on a delay buffer 1515, a directory on the delaybuffer 1515 provides a cross reference between the network address ofthe app/game server 1521-1525 that is the source of the delayedvideo/audio and the location on the delay buffer 1515 where the delayedvideo/audio can be found.

Live, Instantly-Viewable, Instantly-Playable Games

App/game servers 1521-1525 may not only be used for running a givenapplication or video game for a user, but they may also be used forcreating the user interface applications for the hosting service 210that supports navigation through hosting service 210 and other features.A screen shot of one such user interface application is shown in FIG. 16, a “Game Finder” screen. This particular user interface screen allows auser to watch 15 games that are being played live (or delayed) by otherusers. Each of the “thumbnail” video windows, such as 1600 is a livevideo window in motion showing one the video from one user's game. Theview shown in the thumbnail may be the same view that the user isseeing, or it may be a delayed view (e.g., if a user is playing a combatgame, a user may not want other users to see where she is hiding and shemay choose to delay any view of her gameplay by a period of time, say 10minutes). The view may also be a camera view of a game that is differentfrom any user's view. Through menu selections (not shown in thisillustration), a user may choose a selection of games to view at once,based on a variety of criteria. As a small sampling of exemplarychoices, the user may select a random selection of games (such as thoseshown in FIG. 16 ), all of one kind of games (all being played bydifferent players), only the top-ranked players of a game, players at agiven level in the game, or lower-ranked players (e.g., if the player islearning the basics), players who are “buddies” (or are rivals), gamesthat have the most number of viewers, etc.

Note that generally, each user will decide whether the video from his orher game or application can be viewed by others and, if so, whichothers, and when it may be viewed by others, whether it is only viewablewith a delay.

The app/game server 1521-1525 that is generating the user interfacescreen shown in FIG. 16 acquires the 15 video/audio feeds by sending amessage to the app/game server 1521-1525 for each user whose game it isrequesting from. The message is sent through the inbound routing 1502 oranother network. The message will include the size and format of thevideo/audio requested, and will identify the user viewing the userinterface screen. A given user may choose to select “privacy” mode andnot permit any other users to view video/audio of his game (either fromhis point of view or from another point of view), or as described in theprevious paragraph, a user may choose to allow viewing of video/audiofrom her game, but delay the video/audio viewed. A user app/game server1521-1525 receiving and accepting a request to allow its video/audio tobe viewed will acknowledge as such to the requesting server, and it willalso notify the shared hardware compression 1530 of the need to generatean additional compressed video stream in the requested format or screensize (assuming the format and screen size is different than one alreadybeing generated), and it will also indicate the destination for thecompressed video (i.e., the requesting server). If the requestedvideo/audio is only delayed, then the requesting app/game server1521-1525 will be so notified, and it will acquire the delayedvideo/audio from a delay buffer 1515 by looking up the video/audio'slocation in the directory on the delay buffer 1515 and the networkaddress of the app/game server 1521-1525 that is the source of thedelayed video/audio. Once all of these requests have been generated andhandled, up to 15 live thumbnail-sized video streams will be routed fromthe outbound routing 1540 to the inbound routing 1502 to the app/gameserver 1521-1525 generating the user interface screen, and will bedecompressed and displayed by the server. Delayed video/audio streamsmay be in too large a screen size, and if so, the app/game server1521-1525 will decompress the streams and scale down the video streamsto thumbnail size. In one embodiment, requests for audio/video are sentto (and managed by) a central “management” service similar to thehosting service control system of FIG. 4 a (not shown in FIG. 15 ) whichthen redirects the requests to the appropriate app/game server1521-1525. Moreover, in one embodiment, no request may be requiredbecause the thumbnails are “pushed” to the clients of those users thatallow it.

The audio from 15 games all mixed simultaneously might create acacophony of sound. The user may choose to mix all of the soundstogether in this way (perhaps just to get a sense of the “din” createdby all the action being viewed), or the user may choose to just listento the audio from one game at a time. The selection of a single game isaccomplished by moving the yellow selection box 1601 to a given game(the yellow box movement can be accomplished by using arrow keys on akeyboard, by moving a mouse, by moving a joystick, or by pushingdirectional buttons on another device such as a mobile phone). Once asingle game is selected, just the audio from that game plays. Also, gameinformation 1602 is shown. In the case of this game, for example, thepublisher logo (“EA”) and the game logo, “Need for Speed Carbon” and anorange horizontal bar indicates in relative terms the number of peopleplaying or viewing the game at that particular moment (many, in thiscase, so the game is “Hot”). Further “Stats” are provided, indicatingthat there are 145 players actively playing 80 different instantiationsof the Need for Speed Game (i.e., it can be played either by anindividual player game or multiplayer game), and there are 680 viewers(of which this user is one). Note that these statistics (and otherstatistics) are collected by hosting service control system 401 and arestored on RAID arrays 1511-1512, for keeping logs of the hosting service210 operation and for appropriately billing users and paying publisherswho provide content. Some of the statistics are recorded due to actionsby the service control system 401, and some are reported to the servicecontrol system 401 by the individual app/game server 1521-1525. Forexample, the app/game server 1521-1525 running this Game Finderapplication sends messages to the hosting service control system 401when games are being viewed (and when they are ceased to be viewed) sothat it may update the statistics of how many games are in view. Some ofthe statistics are available for user interface applications such asthis Game Finder application.

If the user clicks an activation button on their input device, they willsee the thumbnail video in the yellow box zoom up while it remains liveto full screen size. This effect is shown in process in FIG. 17 . Notethat video window 1700 has grown in size. To implement this effect, theapp/game server 1521-1525 requests from the app/game server 1521-1525running the game selected to have a copy of the video stream for a fullscreen size (at the resolution of the user's display device 422) of thegame routed to it. The app/game server 1521-1525 running the gamenotifies the shared hardware compressor 1530 that a thumbnail-sized copyof the game is no longer needed (unless another app/game server1521-1525 requires such a thumbnail), and then it directs it to send afull-screen size copy of the video to the app/game server 1521-1525zooming the video. The user playing the game may or may not have adisplay device 422 that is the same resolution as that of the userzooming up the game. Further, other viewers of the game may or may nothave display devices 422 that are the same resolution as the userzooming up the game (and may have different audio playback means, e.g.,stereo or surround sound). Thus, the shared hardware compressor 1530determines whether a suitable compressed video/audio stream is alreadybeing generated that meets the requirements of the user requesting thevideo/audio stream and if one does exist, it notifies the outboundrouting 1540 to route a copy of the stream to the app/game server1521-1525 zooming the video, and if not compresses another copy of thevideo that is suitable for that user and instructs the outbound routingto send the stream back to the inbound routing 1502 and the app/gameserver 1521-1525 zooming the video. This server, now receiving a fullscreen version of the selected video will decompress it and graduallyscale it up to full size.

FIG. 18 illustrates how the screen looks after the game has completelyzoomed up to full screen and the game is shown at the full resolution ofthe user's display device 422 as indicated by the image pointed to byarrow 1800. The app/game server 1521-1525 running the game finderapplication sends messages to the other app/game servers 1521-1525 thathad been providing thumbnails that they are no longer needed andmessages to the hosting service control server 401 that the other gamesare no longer being viewed. At this point the only display it isgenerating is an overlay 1801 at the top of the screen which providesinformation and menu controls to the user. Note that as this game hasprogressed, the audience has grown to 2,503 viewers. With so manyviewers, there are bound to be many viewers with display devices 422that have the same or nearly the resolution (each app/game server1521-1525 has the ability to scale the video for adjusting the fitting).

Because the game shown is a multiplayer game, the user may decide tojoin the game at some point. The hosting service 210 may or may notallow the user to join the game for a variety of reasons. For example,the user may have to pay to play the game and choose not to, the usermay not have sufficient ranking to join that particular game (e.g., itwould not be competitive for the other players), or the user's Internetconnection may not have low enough latency to allow the user to play(e.g., there is not a latency constraint for viewing games, so a gamethat is being played far away (indeed, on another continent) can beviewed without latency concerns, but for a game to be played, thelatency must be low enough for the user to (a) enjoy the game, and (b)be on equal footing with the other players who may have lower latencyconnections). If the user is permitted to play, then app/game server1521-1525 that had been providing the Game Finder user interface for theuser will request that the hosting service control server 401 initiate(i.e., locate and start up) an app/game server 1521-1525 that issuitably configured for playing the particular game to load the gamefrom a RAID array 1511-1512, and then the hosting service control server401 will instruct the inbound routing 1502 to transfer the controlsignals from the user to the app/game game server now hosting the gameand it will instruct the shared hardware compression 1530 to switch fromcompressing the video/audio from the app/game server that had beenhosting the Game Finder application to compressing the video/audio fromthe app/game server now hosting the game. The vertical sync of the GameFinder app/game service and the new app/game server hosting the game arenot synchronized, and as a result there is likely to be a timedifference between the two syncs. Because the shared video compressionhardware 1530 will begin compressing video upon an app/game server1521-1525 completing a video frame, the first frame from the new servermay be completed sooner than a full frame time of the old server, whichmay be before the prior compressed frame completing its transmission(e.g., consider transmit time 992 of FIG. 9 b : if uncompressed frame 3963 were completed half a frame time early, it would impinge upon thetransmit time 992). In such a situation the shared video compressionhardware 1530 will ignore the first frame from the new server (e.g.,like Frame 4 964 is ignored 974), and the client 415 will hold the lastframe from the old server an extra frame time, and the shared videocompression hardware 1530 will begin compressing the next frame timevideo from the new app/game server hosting the game. Visually, to theuser, the transition from one app/game server to the other will beseamless. The hosting service control server 401 will then notifyapp/game game server 1521-1525 that had been hosting the Game Finder toswitch to an idle state, until it is needed again.

The user then is able to play the game. And, what is exceptional is thegame will play perceptually instantly (since it will have loaded ontothe app/game game server 1521-1525 from a RAID array 1511-1512 atgigabit/second speed), and the game will be loaded onto a server exactlysuited for the game together with an operating system exactly configuredfor the game with the ideal drivers, registry configuration (in the caseof Windows), and with no other applications running on the server thatmight compete with the game's operation.

Also, as the user progresses through the game, each of the segments ofthe game will load into the server at gigabit/second speed (i.e., 1gigabyte loads in 8 seconds) from the RAID array 1511-1512, and becauseof the vast storage capacity of the RAID array 1511-1512 (since it is ashared resource among many users, it can be very large, yet still becost effective) geometry setup or other game segment setup can bepre-computed and stored on the RAID array 1511-1512 and loaded extremelyrapidly. Moreover, because the hardware configuration and computationalcapabilities of each app/game server 1521-1525 is known, pixel andvertex shaders can be pre-computed.

Thus, the game will start up almost instantly, it will run in an idealenvironment, and subsequent segments will load almost instantly.

But, beyond these advantages, the user will be able to view othersplaying the game (via the Game Finder, previously described and othermeans) and both decide if the game is interesting, and if so, learn tipsfrom watching others. And, the user will be able to demo the gameinstantly, without having to wait for a large download and/orinstallation, and the user will be able to play the game instantly,perhaps on a trial basis for a smaller fee, or on a longer term basis.And, the user will be able to play the game on a Windows PC, aMacintosh, on a television set, at home, when traveling, and even on amobile phone, with a low enough latency wireless connection. And, thiscan all be accomplished without ever physically owning a copy of thegame.

As mentioned previously, the user can decide not allow his gameplay tobe viewable by others, to allow his game to be viewable after a delay,to allow his game to be viewable by selected users, or to allow his gameto be viewable by all users. Regardless, the video/audio will be stored,in one embodiment, for 15 minutes in a delay buffer 1515, and the userwill be able to “rewind” and view his prior game play, and pause, playit back slowly, fast forward, etc., just as he would be able to do hadhe been watching TV with a Digital Video Recorder (DVR). Although inthis example, the user is playing a game, the same “DVR” capability isavailable if the user is using an application. This can be helpful inreviewing prior work and in other applications as detailed below.Further, if the game was designed with the capability of rewinding basedon utilizing game state information, such that the camera view can bechanged, etc., then this “3D DVR” capability will also be supported, butit will require the game to be designed to support it. The “DVR”capability using a delay buffer 1515 will work with any game orapplication, limited of course, to the video that was generated when thegame or application was used, but in the case of games with 3D DVRcapability, the user can control a “fly through” in 3D of a previouslyplayed segment, and have the delay buffer 1515 record the resultingvideo and have the game state of the game segment record. Thus, aparticular “fly-through” will be recorded as compressed video, but sincethe game state will also be recorded, a different fly-through will bepossible at a later date of the same segment of the game.

As described below, users on the hosting service 210 will each have aUser Page, where they can post information about themselves and otherdata. Among of the things that users will be able to post are videosegments from game play that they have saved. For example, if the userhas overcome a particularly difficult challenge in a game, the user can“rewind” to just before the spot where they had their greataccomplishment in the game, and then instruct the hosting service 210 tosave a video segment of some duration (e.g., 30 seconds) on the user'sUser Page for other users to watch. To implement this, it is simply amatter of the app/game server 1521-1525 that the user is using toplayback the video stored in a delay buffer 1515 to a RAID array1511-1512 and then index that video segment on the user's User Page.

If the game has the capability of 3D DVR, as described above, then thegame state information required for the 3D DVR can also be recorded bythe user and made available for the user's User Page.

In the event that a game is designed to have “spectators” (i.e., usersthat are able to travel through the 3D world and observe the actionwithout participating in it) in addition to active players, then theGame Finder application will enable users to join games as spectators aswell as players. From an implementation point of view, there is nodifference to the hosting system 210 to if a user is a spectator insteadof an active player. The game will be loaded onto an app/game server1521-1525 and the user will be controlling the game (e.g., controlling avirtual camera that views into the world). The only difference will bethe game experience of the user.

Multiple User Collaboration

Another feature of the hosting service 210 is the ability to formultiple users to collaborate while viewing live video, even if usingwidely disparate devices for viewing. This is useful both when playinggames and when using applications.

Many PCs and mobile phones are equipped with video cameras and have thecapability to do real-time video compression, particularly when theimage is small. Also, small cameras are available that can be attachedto a television, and it is not difficult to implement real-timecompression either in software or using one of many hardware compressiondevices to compress the video. Also, many PCs and all mobile phones havemicrophones, and headsets are available with microphones.

Such cameras and/or microphones, combined with local video/audiocompression capability (particularly employing the low latency videocompression techniques described herein) will enable a user to transmitvideo and/or audio from the user premises 211 to the hosting service210, together with the input device control data. When such techniquesare employed, then a capability illustrated in FIG. 19 is achievable: auser can have his video and audio 1900 appear on the screen withinanother user's game or application. This example is a multiplayer game,where teammates collaborate in a car race. A user's video/audio could beselectively viewable/hearable only by their teammates. And, since therewould be effectively no latency, using the techniques described abovethe players would be able to talk or make motions to each other inreal-time without perceptible delay.

This video/audio integration is accomplished by having the compressedvideo and/or audio from a user's camera/microphone arrive as inboundinternet traffic 1501. Then the inbound routing 1502 routes the videoand/or audio to the app/game game servers 1521-1525 that are permittedto view/hear the video and/or audio. Then, the users of the respectiveapp/game game servers 1521-1525 that choose to use the video and/oraudio decompress it and integrate as desired to appear within the gameor application, such as illustrated by 1900.

The example of FIG. 19 shows how such collaboration is used in a game,but such collaboration can be an immensely powerful tool forapplications. Consider a situation where a large building is beingdesigned for New York city by architects in Chicago for a real estatedeveloper based in New York, but the decision involves a financialinvestor who is traveling and happens to be in an airport in Miami, anda decision needs to be made about certain design elements of thebuilding in terms of how it fits in with the buildings near it, tosatisfy both the investor and the real estate developer. Assume thearchitectural firm has a high resolution monitor with a camera attachedto a PC in Chicago, the real estate developer has a laptop with a camerain New York, and the investor has a mobile phone with a camera in Miami.The architectural firm can use the hosting service 210 to host apowerful architectural design application that is capable of highlyrealistic 3D rendering, and it can make use of a large database of thebuildings in New York City, as well as a database of the building underdesign. The architectural design application will execute on one, or ifit requires a great deal of computational power on several, of theapp/game servers 1521-1525. Each of the 3 users at disparate locationswill connect to the hosting service 210, and each will have asimultaneous view of the video output of the architectural designapplication, but it will be will appropriately sized by the sharedhardware compression 1530 for the given device and network connectioncharacteristics that each user has (e.g., the architectural firm may seea 2560×1440 60 fps display through a 20 Mbps commercial Internetconnection, the real estate developer in New York may see a 1280×720 60fps image over a 6 Mbps DSL connection on his laptop, and the investormay see a 320×180 60 fps image over a 250 Kbps cellular data connectionon her mobile phone. Each party will hear the voice of the other parties(the conference calling will be handled by any of many widely availableconference calling software package in the app/game server(s) 1521-1525)and, through actuation of a button on a user input device, a user willbe able to make video appear of themselves using their local camera. Asthe meeting proceeds, the architects will be able to show what the buildlooks like as they rotate it and fly by it next to the other building inthe area, with extremely photorealistic 3D rendering, and the same videowill be visible to all parties, at the resolution of each party'sdisplay device. It won't matter that none of the local devices used byany party is incapable of handling the 3D animation with such realism,let alone downloading or even storing the vast database required torender the surrounding buildings in New York City. From the point ofview of each of the users, despite the distance apart, and despite thedisparate local devices they simply will have a seamless experience withan incredible degree of realism. And, when one party wants their face tobe seen to better convey their emotional state, they can do so. Further,if either the real estate develop or the investor want to take controlof the architectural program and use their own input device (be it akeyboard, mouse, keypad or touch screen), they can, and it will respondwith no perceptual latency (assuming their network connection does nothave unreasonable latency). For example, in the case of the mobilephone, if the mobile phone is connected to a WiFi network at theairport, it will have very low latency. But if it is using the cellulardata networks available today in the US, it probably will suffer from anoticeable lag. Still, for most of the purposes of the meeting, wherethe investor is watching the architects control the building fly-by orfor talking of video teleconferencing, even cellular latency should beacceptable.

Finally, at the end of the collaborative conference call, the realestate developer and the investor will have made their comments andsigned off from the hosting service, the architectural firm will be ableto “rewind” the video of the conference that has been recorded on adelay buffer 1515 and review the comments, facial expressions and/oractions applied to the 3D model of the building made during the meeting.If there are particular segments they want to save, those segments ofvideo/audio can be moved from delay buffer 1515 to a RAID array1511-1512 for archival storage and later playback.

Also, from a cost perspective, if the architects only need to use thecomputation power and the large database of New York City for a 15minute conference call, they need only pay for the time that theresources are used, rather than having to own high powered workstationsand having to purchase an expensive copy of a large database.

Video-Rich Community Services

The hosting service 210 enables an unprecedented opportunity forestablishing video-rich community services on the Internet. FIG. 20shows an exemplary User Page for a game player on the hosting service210. As with the Game Finder application, the User Page is anapplication that runs on one of the app/game servers 1521-1525. All ofthe thumbnails and video windows on this page show constantly movingvideo (if the segments are short, they loop).

Using a video camera or by uploading video, the user (whose username is“KILLHAZARD”) is able to post a video of himself 2000 that other userscan view. The video is stored on a RAID array 1511-1512. Also, whenother users come to KILLHAZARD's User Page, if KILLHAZARD is using thehosting service 210 at the time, live video 2001 of whatever he is doing(assuming he permits users viewing his User Page to watch him) will beshown. This will be accomplished by app/game server 1521-1525 hostingthe User Page application requesting from the service control system 401whether KILLHAZARD is active and if so, the app/game server 1521-1525 heis using. Then, using the same methods used by the Game Finderapplication, a compressed video stream in a suitable resolution andformat will be sent to the app/game server 1521-1525 running the UserPage application and it will be displayed. If a user selects the windowwith KILLHAZARD's live gameplay, and then appropriately clicks on theirinput device, the window will zoom up (again using the same methods asthe Game Finder applications, and the live video will fill the screen,at the resolution of the watching user's display device 422, appropriatefor the characteristics of the watching user's Internet connection.

A key advantage of this over prior art approaches is the user viewingthe User Page is able to see a game played live that the user does notown, and may very well not have a local computer or game console capableof playing the game. It offers a great opportunity for the user to seethe user shown in the User Page “in action” playing games, and it is anopportunity to learn about a game that the viewing user might want totry or get better at.

Camera-recorded or uploaded video clips from KILLHAZARD's buddies 2002are also shown on the User Page, and underneath each video clip is textthat indicates whether the buddy is online playing a game (e.g.,six_shot is playing the game “Eragon” and MrSnuggles99 is Offline,etc.). By clicking on a menu item (not shown) the buddy video clipsswitch from showing recorded or uploaded videos to live video of whatthe buddies who are currently playing games on the hosting service 210are doing at that moment in their games. So, it becomes a Game Findergrouping for buddies. If a buddy's game is selected and the user clickson it, it will zoom up to full screen, and the user will be able towatch the game played full screen live.

Again, the user viewing the buddy's game does not own a copy of thecame, nor the local computing/game console resources to play the game.The game viewing is effectively instantaneous.

As previously described above, when a user plays a game on the hostingservice 210, the user is able to “rewind” the game and find a videosegment he wants to save, and then saves the video segment to his UserPage. These are called “Brag Clips”. The video segments 2003 are allBrag Clips 2003 saved by KILLHAZARD from previous games that he hasplayed. Number 2004 shows how many times a Brag Clip has been viewed,and when the Brag Clip is viewed, users have an opportunity to ratethem, and the number of orange keyhole-shaped icons 2005 indicate howhigh the rating is. The Brag Clips 2003 loop constantly when a userviews the User Page, along with the rest of the video on the page. Ifthe user selects and clicks on one of the Brag Clips 2003, it zooms upto present the Brag Clip 2003, along with DVR controls to allow the clipto be played, paused, rewound, fast-forwarded, stepped through, etc.

The Brag Clip 2003 playback is implemented by the app/game server1521-1525 loading the compressed video segment stored on a RAID array1511-1512 when the user recorded the Brag Clip and decompressing it andplaying it back.

Brag Clips 2003 can also be “3D DVR” video segments (i.e., a game statesequence from the game that can be replayed and allows the user tochange the camera viewpoint) from games that support such capability. Inthis case the game state information is stored, in addition to acompressed video recording of the particular “fly through” the user madewhen the game segment was recorded. When the User Page is being viewed,and all of the thumbnails and video windows are constantly looping, a 3DDVR Brag Clip 2003 will constantly loop the Brag Clip 2003 that wasrecorded as compressed video when the user recorded the “fly through” ofthe game segment. But, when a user selects a 3D DVR Brag Clip 2003 andclicks on it, in addition to the DVR controls to allow the compressedvideo Brag Clip to be played, the user will be able to click on a buttonthat gives them 3D DVR capability for the game segment. They will beable to control a camera “fly through” during the game segment on theirown, and, if they wish (and the user who owns the user page so allowsit) they will be able to record an alternative Brag Clip “fly through”in compressed video form will then be available to other viewers of theuser page (either immediately, or after the owner of the user page has achance to the review the Brag Clip).

This 3D DVR Brag Clip 2003 capability is enabled by activating the gamethat is about to replay the recorded game state information on anotherapp/game server 1521-1525. Since the game can be activated almostinstantaneously (as previously described) it is not difficult toactivate it, with its play limited to the game state recorded by theBrag Clip segment, and then allow the user to do a “fly through” with acamera while recording the compressed video to a delay buffer 1515. Oncethe user has completed doing the “fly through” the game is deactivated.

From the user's point of view, activating a “fly through” with a 3D DVRBrag Clip 2003 is no more effort than controlling the DVR controls of alinear Brag Clip 2003. They may know nothing about the game or even howto play the game. They are just a virtual camera operator peering into a3D world during a game segment recorded by another.

Users will also be able to overdub their own audio onto Brag Clips thatis either recorded from microphones or uploaded. In this way, Brag Clipscan be used to create custom animations, using characters and actionsfrom games. This animation technique is commonly known as “machinima”.

As users progress through games, they will achieve differing skilllevels. The games played will report the accomplishments to the servicecontrol system 401, and these skill levels will be shown on User Pages.

Interactive Animated Advertisements

Online advertisements have transitioned from text, to still images, tovideo, and now to interactive segments, typically implemented usinganimation thin clients like Adobe Flash. The reason animation thinclients are used is that users typically have little patience to bedelayed for the privilege of have a product or service pitched to them.Also, thin clients run on very low-performance PCs and as such, theadvertiser can have a high degree of confidence that the interactive adwill work properly. Unfortunately, animation thin clients such as AdobeFlash are limited in the degree of interactivity and the duration of theexperience (to mitigate download time).

FIG. 21 illustrates an interactive advertisement where the user is toselect the exterior and interior colors of a car while the car rotatesaround in a showroom, while real-time ray tracing shows how the carlooks. Then the user chooses an avatar to drive the car, and then theuser can take the car for a drive either on a race track, or through anexotic locale such as Monaco. The user can select a larger engine, orbetter tires, and then can see how the changed configuration affects theability of the car to accelerate or hold the road.

Of course, the advertisement is effectively a sophisticated 3D videogame. But for such an advertisement to be playable on a PC or a videogame console it would require perhaps a 100 MB download and, in the caseof the PC, it might require the installation of special drivers, andmight not run at all if the PC lacks adequate CPU or GPU computingcapability. Thus, such advertisements are impractical in prior artconfigurations.

In the hosting service 210, such advertisements launch almost instantly,and run perfectly, no matter what the user's client 415 capabilitiesare. So, they launch more quickly than thin client interactive ads, arevastly richer in the experience, and are highly reliable.

Streaming Geometry During Real-Time Animation

RAID array 1511-1512 and the inbound routing 1502 can provide data ratesthat are so fast and with latencies so low that it is possible to designvideo games and applications that rely upon the RAID array 1511-1512 andthe inbound routing 1502 to reliably deliver geometry on-the-fly in themidst of game play or in an application during real-time animation(e.g., a fly-through with a complex database.

With prior art systems, such as the video game system shown in FIG. 1 ,the mass storage devices available, particularly in practical homedevices, are far too slow to stream geometry in during game play exceptin situations where the required geometry was somewhat predictable. Forexample, in a driving game where there is a specified roadway, geometryfor buildings that are coming into view can be reasonable well predictedand the mass storage devices can seek in advance to the location wherethe upcoming geometry is located.

But in a complex scene with unpredictable changes (e.g., in a battlescene with complex characters all around) if RAM on the PC or video gamesystem is completely filled with geometry for the objects currently inview, and then the user suddenly turns their character around to viewwhat is behind their character, if the geometry has not been pre-loadedinto RAM, then there may be a delay before it can be displayed.

In the hosting service 210, the RAID arrays 1511-1512 can stream data inexcess of Gigabit Ethernet speed, and with a SAN network, it is possibleto achieve 10 gigabit/second speed over 10 Gigabit Ethernet or overother network technologies. 10 gigabits/second will load a gigabyte ofdata in less that a second. In a 60 fps frame time (16.67 ms),approximately 170 megabits (21 MB) of data can be loaded. Rotatingmedia, of course, even in a RAID configuration will still incurlatencies greater than a frame time, but Flash-based RAID storage willeventually be as large as rotating media RAID arrays and will not incursuch high latency. In one embodiment, massive RAM write-through cachingis used to provide very low latency access.

Thus, with sufficiently high network speed, and sufficiently low enoughlatency mass storage, geometry can be streamed into app/game gameservers 1521-1525 as fast as the CPUs and/or GPUs can process the 3Ddata. So, in the example given previously, where a user turns theircharacter around suddenly and looks behind, the geometry for all of thecharacters behind can be loaded before the character completes therotation, and thus, to the user, it will seem as if he or she is in aphotorealistic world that is as real as live action.

As previously discussed, one of the last frontiers in photorealisticcomputer animation is the human face, and because of the sensitivity ofthe human eye to imperfections, the slightest error from a photorealface can result in a negative reaction from the viewer. FIG. 22 showshow a live performance captured using Contour™ Reality CaptureTechnology (subject of co-pending applications: “Apparatus and methodfor capturing the motion of a performer,” Ser. No. 10/942,609, FiledSep. 15, 2004; “Apparatus and method for capturing the expression of aperformer,” Ser. No. 10/942,413 Filed Sep. 15, 2004; “Apparatus andmethod for improving marker identification within a motion capturesystem,” Ser. No. 11/066,954, Filed Feb. 25, 2005; “Apparatus and methodfor performing motion capture using shutter synchronization,” Ser. No.11/077,628, Filed Mar. 10, 2005; “Apparatus and method for performingmotion capture using a random pattern on capture surfaces,” Ser. No.11/255,854, Filed Oct. 20, 2005; “System and method for performingmotion capture using phosphor application techniques,” Ser. No.11/449,131, Filed Jun. 7, 2006; “System and method for performing motioncapture by strobing a fluorescent lamp,” Ser. No. 11/449,043, Filed Jun.7, 2006; “System and method for three dimensional capture of stop-motionanimated characters,” Ser. No. 11/449,127, Filed Jun. 7, 2006″, each ofwhich is assigned to the assignee of the present CIP application)results in a very smooth captured surface, then a high polygon-counttracked surface (i.e., the polygon motion follows the motion of the faceprecisely). Finally, when the video of the live performance is mapped onthe tracked surface to produce a textured surface, a photoreal result isproduced.

Although current GPU technology is able to render the number of polygonsin the tracked surface and texture and light the surface in real-time,if the polygons and textures are changing every frame time (which willproduce the most photoreal results) it will quickly consume all theavailable RAM of a modern PC or video game console.

Using the streaming geometry techniques described above, it becomespractical to continuously feed geometry into the app/game game servers1521-1525 so that they can animate photoreal faces continuously,allowing the creation of video games with faces that are almostindistinguishable from live action faces.

Integration of Linear Content with Interactive Features

Motion pictures, television programming and audio material(collectively, “linear content” is widely available to home and officeusers in many forms. Linear content can be acquired on physical media,like CD, DVD, HD-DVD and Blu-ray media. It also can be recorded by DVRsfrom satellite and cable TV broadcast. And, it is available aspay-per-view (PPV) content through satellite and cable TV and asvideo-on-demand (VOD) on cable TV.

Increasingly linear content is available through the Internet, both asdownloaded and as streaming content. Today, there really is not oneplace to go to experience all of the features associated with linearmedia. For example, DVDs and other video optical media typically haveinteractive features not available elsewhere, like director'scommentaries, “making of” featurettes, etc. Online music sites havecover art and song information generally not available on CDs, but notall CDs are available online. And Web sites associating with televisionprogramming often have extra features, blogs and sometimes comments fromthe actors or creative staff.

Further, with many motion pictures or sports events, there are oftenvideo games that are released (in the case of motion pictures) oftentogether with the linear media or (in the case of sports) may be closelytied to real-world events (e.g., the trading of players).

Hosting service 210 is well suited for the delivery of linear content inlinking together the disparate forms of related content. Certainly,delivering motion pictures is no more challenging that delivering highlyinteractive video games, and the hosting service 210 is able to deliverlinear content to a wide range of devices, in the home or office, or tomobile devices. FIG. 23 shows an exemplary user interface page forhosting service 210 that shows a selection of linear content.

But, unlike most linear content delivery system, hosting service 210 isalso able to deliver related interactive components (e.g., the menus andfeatures on DVDs, the interactive overlays on HD-DVDs, and the AdobeFlash animation (as explained below) on Web sites. Thus, the clientdevice 415 limitations no longer introduce limitations as to whichfeatures are available.

Further, the hosting system 210 is able to link together linear contentwith video game content dynamically, and in real-time. For example, if auser is watching a Quidditch match in a Harry Potter movie, and decidesshe would like to try playing Quidditch, she can just click a button andthe movie will pause and immediately she will be transported to theQuidditch segment of a Harry Potter video game. After playing theQuidditch match, another click of a button, and the movie will resumeinstantly.

With photoreal graphics and production technology, where thephotographically-captured video is indistinguishable from the liveaction characters, when a user makes a transition from a Quidditch gamein a live action movie to a Quidditch game in a video game on a hostingservice as described herein, the two scenes are virtuallyindistinguishable. This provides entirely new creative options fordirectors of both linear content and interactive (e.g., video game)content as the lines between the two worlds become indistinguishable.

Utilizing the hosting service architecture shown in FIG. 14 the controlof the virtual camera in a 3D movie can be offered to the viewer. Forexample, in a scene that takes place within a train car, it would bepossible to allow the viewer to control the virtual camera and lookaround the car while the story progresses. This assumes that all of the3D objects (“assets”) in the car are available as well as an adequate alevel of computing power capable of rendering the scenes in real-time aswell as the original movie.

And even for non-computer generated entertainment, there are veryexciting interactive features that can be offered. For example, the 2005motion picture “Pride and Prejudice” had many scenes in ornate oldEnglish mansions. For certain mansion scenes, the user may pause thevideo and then control the camera to take a tour of the mansion, orperhaps the surrounding area. To implement this, a camera could becarried through the mansion with a fish-eye lens as it keeps track ofits position, much like prior art Apple, Inc. QuickTime VR isimplemented. The various frames would then be transformed so the imagesare not distorted, and then stored on RAID array 1511-1512 along withthe movie, and played back when the user chooses to go on a virtualtour.

With sports events, a live sports event, such as a basketball game, maybe streamed through the hosting service 210 for users to watch, as theywould for regular TV. After users watched a particular play, a videogame of the game (eventually with basketball players looking asphotoreal as the real players) could come up with the players startingin the same position, and the users (perhaps each taking control of oneplayer) could redo the play to see if they could do better than theplayers.

The hosting service 210 described herein is extremely well-suited tosupport this futuristic world because it is able to bring to bearcomputing power and mass storage resources that are impractical toinstall in a home or in most office settings, and also it's computingresources are always up-to-date, with the latest computing hardwareavailable, whereas in a home setting, there will always be homes witholder generation PCs and video games. And, in the hosting service 210,all of this computing complexity is hidden from the user, so even thoughthey may be using very sophisticated systems, from the user's point ofview, it is a simple as changing channels on a television. Further, theusers would be able to access all of the computing power and theexperiences the computing power would bring from any client 415.

Multiplayer Games

To the extent the game is a multiplayer game, then it will be ablecommunicate both to app/game game servers 1521-1525 through the inboundrouting 1502 network and, with a network bridge to the Internet (notshown) with servers or game machines that are not running in the hostingservice 210. When playing multiplayer games with computers on thegeneral Internet, then the app/game game servers 1521-1525 will have thebenefit of extremely fast access to the Internet (compared to if thegame was running on a server at home), but they will be limited by thecapabilities of the other computers playing the game on slowerconnections, and also potentially limited by the fact that the gameservers on the Internet were designed to accommodate the least commondenominator, which would be home computers on relatively slow consumerInternet connections.

But when a multiplayer game is played entirely within a hosting service210 server center, then a world of difference is achievable. Eachapp/game game server 1521-1525 hosting a game for a user will beinterconnected with other app/game game servers 1521-1525 as well as anyservers that are hosting the central control for the multiplayer gamewith extremely high speed, extremely low latency connectivity and vast,very fast storage arrays. For example, if Gigabit Ethernet is used forthe inbound routing 1502 network, then the app/game game servers1521-1525 will be communicating among each other and communicating toany servers hosting the central control for the multiplayer game atgigabit/second speed with potentially only 1 ms of latency or less.Further, the RAID arrays 1511-1512 will be able to respond very rapidlyand then transfer data at gigabit/second speeds. As an example, if auser customizes a character in terms of look and accoutrements such thatthe character has a large amount of geometry and behaviors that areunique to the character, with prior art systems limited to the gameclient running in the home on a PC or game console, if that characterwere to come into view of another user, the user would have to waituntil a long, slow download completes so that all of the geometry andbehavior data loads into their computer. Within the hosting service 210,that same download could be over Gigabit Ethernet, served from a RAIDarray 1511-1512 at gigabit/second speed. Even if the home user had an 8Mbps Internet connection (which is extremely fast by today's standards),Gigabit Ethernet is 100 times faster. So, what would take a minute overa fast Internet connection, would take less than a second over GigabitEthernet.

Top Player Groupings and Tournaments

The Hosting Service 210 is extremely well-suited for tournaments.Because no game is running in a local client, there is no opportunityfor users to cheat. Also, because of the ability of the output routing1540 to multicast the UDP streams, the Hosting Service is 210 is able tobroadcast the major tournaments to thousands of people in the audienceat once.

In fact, when there are certain video streams that are so popular thatthousands of users are receiving the same stream (e.g., showing views ofa major tournament), it may be more efficient to send the video streamto a Content Delivery Network (CDN) such as Akamai or Limelight for massdistribution to many client devices 415.

A similar level of efficiency can be gained when a CDN is used to showGame Finder pages of top player groupings.

For major tournaments, a live celebrity announcer can be used to providecommentary during certain matches. Although a large number of users willbe watching a major tournament, and relatively small number will beplaying in the tournament. The audio from the celebrity announcer can berouted to the app/game game servers 1521-1525 hosting the users playingin the tournament and hosting any spectator mode copies of the game inthe tournament, and the audio can be overdubbed on top of the gameaudio. Video of a celebrity announcer can be overlaid on the games,perhaps just on spectator views, as well.

Acceleration of Web Page Loading

The World Wide Web its primary transport protocol, Hypertext TransferProtocol (HTTP), were conceived and defined in an era where onlybusinesses had high speed Internet connections, and the consumers whowere online were using dialup modems or ISDN. At the time, the “goldstandard” for a fast connection was a T1 line which provided 1.5 Mbpsdata rate symmetrically (i.e., with equal data rate in both directions).

Today, the situation is completely different. The average homeconnection speed through DSL or cable modem connections in much of thedeveloped world has a far higher downstream data rate than a T1 line. Infact, in some parts of the world, fiber-to-the-curb is bringing datarates as high as 50 to 100 Mbps to the home.

Unfortunately, HTTP was not architected (nor has it been implemented) toeffectively take advantage of these dramatic speed improvements. A website is a collection of files on a remote server. In very simple terms,HTTP requests the first file, waits for the file to be downloaded, andthen requests the second file, waits for the file to be downloaded, etc.In fact, HTTP allows for more than one “open connection”, i.e., morethan one file to be requested at a time, but because of agreed-uponstandards (and a desire to prevent web servers from being overloaded)only very few open connections are permitted. Moreover, because of theway Web pages are constructed, browsers often are not aware of multiplesimultaneous pages that could be available to download immediately(i.e., only after parsing a page does it become apparent that a newfile, like an image, needs to be downloaded). Thus, files on website areessentially loaded one-by-one. And, because of the request-and-responseprotocol used by HTTP, there is roughly (accessing typical web serversin the US) a 100 ms latency associated with each file that is loaded.

With relatively low speed connections, this does not introduce much of aproblem because the download time for the files themselves dominates thewaiting time for the web pages. But, as connection speeds grow,especially with complex web pages, problems begin to arise.

In the example shown in FIG. 24 , a typical commercial website is shown(this particular website was from a major athletic shoe brand). Thewebsite has 54 files on it. The files include HTML, CSS, JPEG, PHP,JavaScript and Flash files, and include video content. A total of 1.5MBytes must be loaded before the page is live (i.e., the user can clickon it and begin to use it). There are a number of reasons for the largenumber of files. For one thing, it is a complex and sophisticatedwebpage, and for another, it is a webpage that is assembled dynamicallybased on the information about the user accessing the page (e.g., whatcountry the user is from, what language, whether the user has madepurchases before, etc.), and depending on all of these factors,different files are downloaded. Still, it is a very typical commercialweb page.

FIG. 24 shows the amount of time that elapses before the web page islive as the connection speed grows. With a 1.5 Mbps connection speed2401, using a conventional web server with a convention web browser, ittakes 13.5 seconds until the web page is live. With a 12 Mbps connectionspeed 2402, the load time is reduced to 6.5 seconds, or about twice asfast. But with a 96 Mbps connection speed 2403, the load time is onlyreduced to about 5.5 seconds. The reason why is because at such a highdownload speed, the time to download the files themselves is minimal,but the latency per file, roughly 100 ms each, still remains, resultingin 54 files*100 ms=5.4 seconds of latency. Thus, no matter how fast theconnection is to the home, this web site will always take at least 5.4seconds until it is live. Another factor is the server-side queuing;every HTTP request is added in the back of the queue, so on a busyserver this will have a significant impact because for every small itemto get from the web server, the HTTP requests needs to wait for itsturn.

One way to solve these issues is to discard or redefine HTTP. Or,perhaps to get the website owner to better consolidate its files into asingle file (e.g., in Adobe Flash format). But, as a practical matter,this company, as well as many others has a great deal of investment intheir web site architecture. Further, while some homes have 12-100 Mbpsconnections, the majority of homes still have slower speeds, and HTTPdoes work well at slow speed.

One alternative is to host web browsers on app/game servers 1521-1525,and host the files for the web servers on the RAID arrays 1511-1512 (orpotentially in RAM or on local storage on the app/game servers 1521-1525hosting the web browsers. Because of the very fast interconnect throughthe inbound routing 1502 (or to local storage), rather than have 100 msof latency per file using HTTP, there will be de minimis latency perfile using HTTP. Then, instead of having the user in her home accessingthe web page through HTTP, the user can access the web page throughclient 415. Then, even with a 1.5 Mbps connection (because this web pagedoes not require much bandwidth for its video), the webpage will be livein less than 1 second per line 2400. Essentially, there will be nolatency before the web browser running on an app/game server 1521-1525is displaying a live page, and there will be no detectable latencybefore the client 415 displays the video output from the web browser. Asthe user mouses around and/or types on the web page, the user's inputinformation will be sent to the web browser running on the app/gameserver 1521-1525, and the web browser will respond accordingly.

One disadvantage to this approach is if the compressor is constantlytransmitting video data, then bandwidth is used, even if the web pagebecomes static. This can be remedied by configuring the compressor toonly transmit data when (and if) the web page changes, and then, onlytransmit data to the parts of the page that change. While there are someweb pages with flashing banners, etc. that are constantly changing, suchweb pages tend to be annoying, and usually web pages are static unlessthere is a reason for something to be moving (e.g., a video clip). Forsuch web pages, it is likely the case the less data will be transmittedusing the hosting service 210 than a conventional web server becauseonly the actual displayed images will be transmitted, no thin clientexecutable code, and no large objects that may never be viewed, such asrollover images.

Thus, using the hosting service 210 to host legacy web pages, web pageload times can be reduces to the point where opening a web page is likechanging channels on a television: the web page is live effectivelyinstantly.

Facilitating Debugging of Games and Applications

As mentioned previously, video games and applications with real-timegraphics are very complex applications and typically when they arereleased into the field they contain bugs. Although software developerswill get feedback from users about bugs, and they may have some means topass back machine state after crashes, it is very difficult to identifyexactly what has caused a game or real-time application to crash or toperform improperly.

When a game or application runs in the hosting service 210, thevideo/audio output of the game or application is constantly recorded ona delay buffer 1515. Further, a watchdog process runs each app/gameserver 1521-1525 which reports regularly to the hosting service controlsystem 401 that the app/game server 1521-1525 is running smoothly. Ifthe watchdog process fails to report in, then the server control system401 will attempt to communicate with the app/game server 1521-1525, andif successful, will collect whatever machine state is available.Whatever information is available, along with the video/audio recordedby the delay buffer 1515 will be sent to the software developer.

Thus, when the game or application software developer gets notificationof a crash from the hosting service 210, it gets a frame-by-frame recordof what led up to the crash. This information can be immensely valuablein tracking down bugs and fixing them.

Note also, that when an app/game server 1521-1525 crashes, the server isrestarted at the most recent restartable point, and a message isprovided to the user apologizing for the technical difficulty.

Resource Sharing and Cost Savings

The system shown in FIGS. 4 a and 4 b provide a variety of benefits forboth end users and game and application developers. For example,typically, home and office client systems (e.g., PCs or game consoles)are only in use for a small percentage of the hours in a week. Accordingto an Oct. 5, 2006 press release by the Nielsen Entertainment “ActiveGamer Benchmark Study”(http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-05-2006/0004446115&EDATE=)active gamers spend on average 14 hours a week playing on video gameconsoles and about 17 hours a week on handhelds. The report also statesthat for all game playing activity (including console, handheld and PCgame playing) Active Gamers average 13 hours a week. Taking intoconsideration the higher figure of console video game playing time,there are 24*7=168 hours in a week, that implies that in an activegamer's home, a video game console is in use only 17/168=10% of thehours of a week. Or, 90% of the time, the video game console is idle.Given the high cost of video game consoles, and the fact thatmanufacturers subsidize such devices, this is a very inefficient use ofan expensive resource. PCs within businesses are also typically usedonly a fraction of the hours of the week, especially non-portabledesktop PCs often required for high-end applications such as AutodeskMaya. Although some businesses operate at all hours and on holidays, andsome PCs (e.g., portables brought home for doing work in the evening)are used at all hours and holidays, most business activities tend tocenter around 9 AM to 5 PM, in a given business' time zone, from Mondayto Friday, less holidays and break times (such as lunch), and since mostPC usage occurs while the user is actively engaged with the PC, itfollows that desktop PC utilization tends to follow these hours ofoperation. If we were to assume that PCs are utilized constantly from 9AM to 5 PM, 5 days a week, that would imply PCs are utilized 40/168=24%of the hours of the week. High-performance desktop PCs are veryexpensive investments for businesses, and this reflects a very low levelof utilization. Schools that are teaching on desktop computers may usecomputers for an even smaller fraction of the week, and although itvaries depending upon the hours of teaching, most teaching occurs duringthe daytime hours from Monday through Friday. So, in general, PCs andvideo game consoles are utilized only a small fraction of the hours ofthe week.

Notably, because many people are working at businesses or at schoolduring the daytime hours of Monday through Friday on non-holidays, thesepeople generally are not playing video games during these hours, and sowhen they do play video games it is generally during other hours, suchas evenings, weekends and on holidays.

Given the configuration of the hosting service shown in FIG. 4 a , theusage patterns described in the above two paragraphs result in veryefficient utilization of resources. Clearly, there is a limit to thenumber of users who can be served by the hosting service 210 at a giventime, particularly if the users are requiring real-time responsivenessfor complex applications like sophisticated 3D video games. But, unlikea video game console in a home or a PC used by a business, whichtypically sits idle most of the time, servers 402 can be re-utilized bydifferent users at different times. For example, a high-performanceserver 402 with high performance dual CPUs and dual GPUs and a largequantity of RAM can be utilized by a businesses and schools from 9 AM to5 PM on non-holidays, but be utilized by gamers playing a sophisticatedvideo game in the evenings, weekends and on holidays. Similarly,low-performance applications can be utilized by businesses and schoolson a low-performance server 402 with a Celeron CPU, no GPU (or a verylow-end GPU) and limited RAM during business hours and a low-performancegame can utilize a low-performance server 402 during non-business hours.

Further, with the hosting service arrangement described herein,resources are shared efficiently among thousands, if not millions, ofusers. In general, online services only have a small percentage of theirtotal user base using the service at a given time. If we consider theNielsen video game usage statistics listed previously, it is easy to seewhy. If active gamers play console games only 17 hours of a week, and ifwe assume that the peak usage time for game is during the typicalnon-work, non-business hours of evenings (5-12 AM, 7*5 days=35hours/week) and weekend (8 AM-12 AM, 16*2=32 hours/week), then there are35+32=65 peak hours a week for 17 hours of game play. The exact peakuser load on the system is difficult to estimate for many reasons: someusers will play during off-peak times, there may be certain day timeswhen there are clustering peaks of users, the peak times can be affectedby the type of game played (e.g., children's games will likely be playedearlier in the evening), etc. But, given that the average number ofhours played by a gamer is far less than the number of hours of the daywhen a gamer is likely to play a game, only a fraction of the number ofusers of the hosting service 210 will be using it at a given time. Forthe sake of this analysis, we shall assume the peak load is 12.5%. Thus,only 12.5% of the computing, compression and bandwidth resources areused at a given time, resulting in only 12.5% of the hardware cost tosupport a given user to play a given level of performance game due toreuse of resources.

Moreover, given that some games and applications require more computingpower than others, resources may be allocated dynamically based on thegame being played or the applications executed by users. So, a userselecting a low-performance game or application will be allocated alow-performance (less expensive) server 402, and a user selecting ahigh-performance game or applications will be allocated ahigh-performance (more expensive) server 402. Indeed, a given game orapplication may have lower-performance and higher-performance sectionsof the game or applications, and the user can be switched from oneserver 402 to another server 402 between sections of the game orapplication to keep the user running on the lowest-cost server 402 thatmeets the game or application's needs. Note that the RAID arrays 405,which will be far faster than a single disk, will be available to evenlow-performance servers 402, that will have the benefit of the fasterdisk transfer rates. So, the average cost per server 402 across all ofthe games being played or applications being used is much less than thecost of the most expensive server 402 that plays the highest performancegame or applications, yet even the low-performance servers 402, willderive disk performance benefits from the RAID arrays 405.

Further, a server 402 in the hosting service 210 may be nothing morethan a PC motherboard without a disk or peripheral interfaces other thana network interface, and in time, may be integrated down to a singlechip with just a fast network interface to the SAN 403. Also, RAIDArrays 405 likely will be shared amongst far many more users than thereare disks, so the disk cost per active user will be far less than onedisk drive. All of this equipment will likely reside in a rack in aenvironmentally-controlled server room environment. If a server 402fails, it can be readily repaired or replaced at the hosting service210. In contrast, a PC or game console in the home or office must be asturdy, standalone appliance that has to be able to survive reasonablewear and tear from being banged or dropped, requires a housing, has atleast one disk drive, has to survive adverse environment conditions(e.g., being crammed into an overheated AV cabinet with other gear),requires a service warranty, has to be packaged and shipped, and is soldby a retailer who will likely collect a retail margin. Further, a PC orgame console must be configured to meet the peak performance of the mostcomputationally-intensive anticipated game or application to be used atsome point in the future, even though lower performance games orapplication (or sections of games or applications) may be played most ofthe time. And, if the PC or console fails, it is an expensive andtime-consuming process (adversely impacting the manufacturer, user andsoftware developer) to get it repaired.

Thus, given that the system shown in FIG. 4 a provides an experience tothe user comparable to that of a local computing resource, for a user inthe home, office or school to experience a given level of computingcapability, it is much less expensive to provide that computingcapability through the architecture shown in FIG. 4 a.

Eliminating the Need to Upgrade

Further, users no longer have to worry about upgrading PCs and/orconsoles to play new games or handle higher performance newapplications. Any game or applications on the hosting service 210,regardless of what type of server 402 is required for that game orapplications, is available to the user, and all games and applicationsrun nearly instantly (i.e., loading rapidly from the RAID Arrays 405 orlocal storage on a servers 402) and properly with the latest updates andbug fixes (i.e., software developers will be able to choose an idealserver configuration for the server(s) 402 that run(s) a given game orapplication, and then configure the server(s) 402 with optimal drivers,and then over time, the developers will be able to provide updates, bugfixes, etc. to all copies of the game or application in the hostingservice 210 at once). Indeed, after the user starts using the hostingservice 210, the user is likely to find that games and applicationscontinue to provide a better experience (e.g., through updates and/orbug fixes) and it may be the case that user discovers a year later thata new game or application is made available on the service 210 that isutilizing computing technology (e.g., a higher-performance GPU) that didnot even exist a year before, so it would have been impossible for theuser to buy the technology a year before that would play the game or runthe applications a year later. Since the computing resource that isplaying the game or running the application is invisible to the user(i.e., from the user's perspective the user is simply selecting a gameor application that begins running nearly instantly—much as if the userhad changed channels on a television), the user's hardware will havebeen “upgraded” without the user even being aware of the upgrade.

Eliminating the Need for Backups

Another major problem for users in businesses, schools and homes arebackups. Information stored in a local PC or video game console (e.g.,in the case of a console, a user's game achievements and ranking) can belost if a disk fails, or if there is an inadvertent erasure. There aremany applications available that provide manual or automatic backups forPCs, and game console state can be uploaded to an online server forbackup, but local backups are typically copied to another local disk (orother non-volatile storage device) which has to be stored somewhere safeand organized, and backups to online services are often limited becauseof the slow upstream speed available through typical low-cost Internetconnections. With the hosting service 210 of FIG. 4 a , the data that isstored in RAID arrays 405 can be configured using prior art RAIDconfiguration techniques well-known to those skilled in the art suchthat if a disk fails, no data will be lost, and a technician at theserver center housing the failed disk will be notified, and then willreplace the disk, which then will be automatically updated so that theRAID array is once again failure tolerant. Further, since all of thedisk drives are near one another and with fast local networks betweenthem through the SAN 403 it is not difficult in a server center toarrange for all of the disk systems to be backed up on a regular basisto secondary storage, which can be either stored at the server center orrelocated offsite. From the point of view of the users of hostingservice 210, their data is simply secure all the time, and they neverhave to think about backups.

Access to Demos

Users frequently want to try out games or applications before buyingthem. As described previously, there are prior art means by which todemo (the verb form of “demo” means to try out a demonstration version,which is also called a “demo”, but as a noun) games and applications,but each of them suffers from limitations and/or inconveniences. Usingthe hosting service 210, it is easy and convenient for users to try outdemos. Indeed, all the user does is select the demo through a userinterface (such as one described below) and try out the demo. The demowill load almost instantly onto a server 402 appropriate for the demo,and it will just run like any other game or application. Whether thedemo requires a very high performance server 402, or a low performanceserver 402, and no matter what type of home or office client 415 theuser is using, from the point of view of the user, the demo will justwork. The software publisher of either the game or application demo willbe able to control exactly what demo the user is permitted to try outand for how long, and of course, the demo can include user interfaceelements that offer the user an opportunity to gain access to a fullversion of the game or application demonstrated.

Since demos are likely to be offered below cost or free of charge, someusers may try to use demos repeated (particularly game demos, which maybe fun to play repeatedly). The hosting service 210 can employ varioustechniques to limit demo use for a given user. The most straightforwardapproach is to establish a user ID for each user and limit the number oftimes a given user ID is allowed to play a demo. A user, however, mayset up multiple user IDs, especially if they are free. One technique foraddressing this problem is to limit the number of times a given client415 is allowed to play a demo. If the client is a standalone device,then the device will have a serial number, and the hosting service 210can limit the number of times a demo can be accessed by a client withthat serial number. If the client 415 is running as software on a PC orother device, then a serial number can be assigned by the hostingservice 210 and stored on the PC and used to limit demo usage, but giventhat PCs can be reprogrammed by users, and the serial number erased orchanged, another option is for the hosting service 210 to keep a recordof the PC network adapter Media Access Control (MAC) address (and/orother machine specific identifiers such as hard-drive serial numbers,etc.) and limit demo usage to it. Given that the MAC addresses ofnetwork adapters can be changed, however, this is not a foolproofmethod. Another approach is to limit the number of times a demo can beplayed to a given IP address. Although IP addresses may be periodicallyreassigned by cable modem and DSL providers, it does not happen inpractice very frequently, and if it can be determined (e.g., bycontacting the ISP) that the IP is in a block of IP addresses forresidential DSL or cable modem accesses, then a small number of demouses can typically be established for a given home. Also, there may bemultiple devices at a home behind a NAT router sharing the same IPaddress, but typically in a residential setting, there will be a limitednumber of such devices. If the IP address is in a block servingbusinesses, then a larger number of demos can be established for abusiness. But, in the end, a combination of all of the previouslymentioned approaches is the best way to limit the number of demos onPCs. Although there may be no foolproof way that a determined andtechnically adept user can be limited in the number of demos playedrepeatedly, creating a large number of barriers can create a sufficientdeterrent such that it's not worth the trouble most PC users to abusethe demo system, and rather they use the demos as they were intended totry out new games and applications.

Benefits to Schools, Businesses and Other Institutions

Significant benefits accrue particularly to businesses, schools andother institutions that utilize the system shown in FIG. 4 a .Businesses and schools have substantial costs associated withinstalling, maintaining and upgrading PCs, particularly when it comes toPCs for running high-performance applications, such a Maya. As statedpreviously, PCs are generally utilized only a fraction of the hours ofthe week, and as in the home, the cost of PC with a given level ofperformance capability is far higher in an office or school environmentthan in a server center environment.

In the case of larger businesses or schools (e.g., large universities),it may be practical for the IT departments of such entities to set upserver centers and maintain computers that are remotely accessed viaLAN-grade connections. A number of solutions exist for remote access ofcomputers over a LAN or through a private high bandwidth connectionbetween offices. For example, with Microsoft's Windows Terminal Server,or through virtual network computing applications like VNC, fromRealVNC, Ltd., or through thin client means from Sun Microsystems, userscan gain remote access to PCs or servers, with a range of quality ingraphics response time and user experience. Further, such self-managedserver centers are typically dedicated for a single business or schooland as such, are unable to take advantage of the overlap of usage thatis possible when disparate applications (e.g., entertainment andbusiness applications) utilize the same computing resources at differenttimes of the week. So, many businesses and schools lack the scale,resources or expertise to set up a server center on their own that has aLAN-speed network connection to each user. Indeed, a large percentage ofschools and businesses have the same Internet connections (e.g., DSL,cable modems) as homes.

Yet such organizations may still have the need for very high-performancecomputing, either on a regular basis or on a periodic basis. Forexample, a small architectural firm may have only a small number ofarchitects, with relatively modest computing needs when doing designwork, but it may require very high-performance 3D computing periodically(e.g., when creating a 3D fly-through of a new architectural design fora client). The system shown in FIG. 4 a is extremely well suited forsuch organizations. The organizations need nothing more than the samesort of network connection that are offered to homes (e.g., DSL, cablemodems) and are typically very inexpensive. They can either utilizeinexpensive PCs as the client 415 or dispense with PCs altogether andutilize inexpensive dedicated devices which simply implement the controlsignal logic 413 and low-latency video decompression 412. These featuresare particularly attractive for schools that may have problems withtheft of PCs or damage to the delicate components within PCs.

Such an arrangement solves a number of problems for such organizations(and many of these advantages are also shared by home users doinggeneral-purpose computing). For one, the operating cost (whichultimately must be passed back in some form to the users in order tohave a viable business) can be much lower because (a) the computingresources are shared with other applications that have different peakusage times during the week, (b) the organizations can gain access to(and incur the cost of) high performance computing resources only whenneeded, (c) the organizations do not have to provide resources forbacking up or otherwise maintaining the high performance computingresources.

Elimination of Piracy

In addition, games, applications, interactive movies, etc, can no longerbe pirated as they are today. Because game is executed at the servicecenter, users are not provided with access to the underlying programcode, so there is nothing to pirate. Even if a user were to copy thesource code, the user would not be able to execute the code on astandard game console or home computer. This opens up markets in placesof the world such as China, where standard video gaming is not madeavailable. The re-sale of used games is also not possible.

For game developers, there are fewer market discontinuities as is thecase today. The hosting service 210 can be gradually updated over timeas gaming requirements change, in contrast to the current situationwhere a completely new generation of technology forces users anddevelopers to upgrade and the game developer is dependent on the timelydelivery of the hardware platform.

Streaming Interactive Video

The above descriptions provide a wide range of applications enabled bythe novel underlying concept of general Internet-based, low-latencystreaming interactive video (which implicitly includes audio togetherwith the video as well, as used herein). Prior art systems that haveprovided streaming video through the Internet only have enabledapplications which can be implemented with high latency interactions.For example, basic playback controls for linear video (e.g. pause,rewind, fast forward) work adequately with high latency, and it ispossible to select among linear video feeds. And, as stated previously,the nature of some video games allow them to be played with highlatency. But the high latency (or low compression ratio) of prior artapproaches for streaming video have severely limited the potentialapplications of streaming video or narrowed their deployments tospecialized network environments, and even in such environments, priorart techniques introduce substantial burdens on the networks. Thetechnology described herein opens the door for the wide range ofapplications possible with low-latency streaming interactive videothrough the Internet, particularly those enabled through consumer-gradeInternet connections.

Indeed, with client devices as small as client 465 of FIG. 4 csufficient to provide an enhanced user experience with an effectivelyarbitrary amount of computing power, arbitrary amount of fast storage,and extremely fast networking amongst powerful servers, it enables a newera of computing. Further, because the bandwidth requirements do notgrow as the computing power of the system grows (i.e., because thebandwidth requirements are only tied to display resolution, quality andframe rate), once broadband Internet connectivity is ubiquitous (e.g.,through widespread low-latency wireless coverage), reliable, and ofsufficiently high bandwidth to meet the needs of the display devices 422of all users, the question will be whether thick clients (such as PCs ormobile phones running Windows, Linux, OSX, etc.) or even thin clients(such as Adobe Flash or Java) are necessary for typical consumer andbusiness applications.

The advent of streaming interactive video results in a rethinking ofassumptions about the structure of computing architectures. An exampleof this is the hosting service 210 server center embodiment shown inFIG. 15 . The video path for delay buffer and/or group video 1550 is afeedback loop where the multicasted streaming interactive video outputof the app/game servers 1521-1525 is fed back into the app/game servers1521-1525 either in real-time via path 1552 or after a selectable delayvia path 1551. This enables a wide range of practical applications (e.g.such as those illustrated in FIGS. 16, 17 and 20 ) that would be eitherimpossible or infeasible through prior art server or local computingarchitectures. But, as a more general architectural feature, whatfeedback loop 1550 provides is recursion at the streaming interactivevideo level, since video can be looped back indefinitely as theapplication requires it. This enables a wide range of applicationpossibilities never available before.

Another key architectural feature is that the video streams areunidirectional UDP streams. This enables effectively an arbitrary degreeof multicasting of streaming interactive video (in contrast, two-waystreams, such as TCP/IP streams, would create increasingly more trafficlogjams on the networks from the back-and-forth communications as thenumber of users increased). Multicasting is an important capabilitywithin the server center because it allows the system to be responsiveto the growing needs of Internet users (and indeed of the world'spopulation) to communicate on a one-to-many, or even a many-to-manybasis. Again, the examples discussed herein, such as FIG. 16 whichillustrates the use of both streaming interactive video recursion andmulticasting are just the tip of a very large iceberg of possibilities.

In one embodiment, the various functional modules illustrated herein andthe associated steps may be performed by specific hardware componentsthat contain hardwired logic for performing the steps, such as anapplication-specific integrated circuit (“ASIC”) or by any combinationof programmed computer components and custom hardware components.

In one embodiment, the modules may be implemented on a programmabledigital signal processor (“DSP”) such as a Texas Instruments' TMS320xarchitecture (e.g., a TMS320C6000, TMS320C5000, . . . etc). Variousdifferent DSPs may be used while still complying with these underlyingprinciples.

Embodiments may include various steps as set forth above. The steps maybe embodied in machine-executable instructions which cause ageneral-purpose or special-purpose processor to perform certain steps.Various elements which are not relevant to these underlying principlessuch as computer memory, hard drive, input devices, have been left outof the figures to avoid obscuring the pertinent aspects.

Elements of the disclosed subject matter may also be provided as amachine-readable medium for storing the machine-executable instructions.The machine-readable medium may include, but is not limited to, flashmemory, optical disks, CD-ROMs, DVD ROMs, RAMs, EPROMs, EEPROMs,magnetic or optical cards, or other type of machine-readable mediasuitable for storing electronic instructions.

It should also be understood that elements of the disclosed subjectmatter may also be provided as a computer program product which mayinclude a machine-readable medium having stored thereon instructionswhich may be used to program a computer (e.g., a processor or otherelectronic device) to perform a sequence of operations. Alternatively,the operations may be performed by a combination of hardware andsoftware. The machine-readable medium may include, but is not limitedto, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks,ROMs, RAMs, EPROMs, EEPROMs, magnet or optical cards, or other type ofmachine-readable medium suitable for storing electronic instructions.

Additionally, although the disclosed subject matter has been describedin conjunction with specific embodiments, numerous modifications andalterations are well within the scope of the present disclosure.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

The invention claimed is:
 1. A computer-implemented method comprising:executing a game application on one or more servers of a data center,wherein the game application is for a game and the game is played by afirst user of a first client device remote to the data center, the oneor more servers working with an encoder for compressing and streaminginteractive video to the first client device and using input from thefirst client device to control interactivity when playing the game, thestreaming of the interactive video is transmitted over the Internet tothe first client device; storing application state data for at leastpart of the game played by the first user using the first client device;and generating video, responsive to input from a second client device,for a camera view into a scene of the game played by the first user;wherein the video is generated using the application state data as inputfor re-executing the game application.
 2. The method of claim 1, whereinthe second client device is of a spectator of the first user.
 3. Themethod of claim 1, wherein the second client device is a remote deviceconnected to a server of the data center, and said second client deviceis provided with functionality to view, pause, and rewind a recording ofthe game played by the first user.
 4. The method of claim 3, whereinsaid rewind is of the recording, and said viewing is of an interactivevideo generated for the first user or a different camera view.
 5. Themethod of claim 1, wherein the second client device is provided withfunctionality to request generation of additional video to viewadditional camera views of the scene of the game played by the firstuser.
 6. The method of claim 1, wherein said camera view is acontrollable fly-through perspective controlled by the second clientdevice.
 7. The method of claim 1, wherein the application state dataenables a plurality of replays with changes in the camera view, where ineach replay provides for controlling a fly-through perspective into thescene.
 8. The method of claim 1, wherein said second client device isprovided with a user interface that enables selection of differentcamera views of the game play of the first user or other users.
 9. Themethod of claim 1, further comprising, enabling the first user tospecify a segment of game play for sharing with one or more spectators.10. The method of claim 9, further comprising, saving the segment forviewing or play on a website, said website being accessible over theInternet by the first user, a second user or other users.
 11. A computerreadable memory having program instructions for executing a gameapplication by a processor, comprising: program instructions forexecuting the game application on one or more servers of a data center,wherein the game application is for a game and the game is played by afirst user of a first client device remote to the data center, the oneor more servers working with an encoder for compressing and streaminginteractive video to the first client device and using input from thefirst client device to control interactivity when playing the game, thestreaming of the interactive video is transmitted over the Internet tothe first client device; program instructions for storing applicationstate data for at least part of the game played by the first user usingthe first client device; and program instructions for generating video,responsive to input from a second client device, for a camera view intoa scene of the game played by the first user; wherein the video isgenerated using the application state data as input for re-executing thegame application.
 12. The compute readable memory of claim 11, whereinthe second client device is of a spectator of the first user.
 13. Thecompute readable memory of claim 11, wherein the second client device isa remote device connected to a server of the data center, and saidsecond client device is provided with functionality to view, pause, andrewind a recording of the game played by the first user.
 14. The computereadable memory of claim 13, wherein said rewind is of the recording,and said viewing is of an interactive video generated for the first useror the different camera view.
 15. The compute readable memory of claim11, wherein the second user device is provided with functionality torequest generation of additional video to view additional camera viewsof the scene of the game played by the first user.
 16. The computereadable memory of claim 11, wherein said camera view is a controllablefly-through perspective controlled by the second client device.
 17. Thecompute readable memory of claim 11, wherein the application state dataenables a plurality of replays with changes in the camera view, where ineach replay provides for controlling a fly-through perspective into thescene.
 18. The compute readable memory of claim 11, wherein said seconduser device is provided with a user interface that enables selection ofdifferent camera views of the game play of the first user or otherusers.
 19. The compute readable memory of claim 11, further comprising,enabling the first user to specify a segment of the game play forsharing with one or more spectators.
 20. The compute readable memory ofclaim 19, further comprising, saving the segment for viewing or play ona website, said website being accessible over the Internet by the firstuser, the second user or other users.